Hand presence sensing at control input device

ABSTRACT

Implementations relate to hand presence sensing at a control input device. In some implementations, a control input device includes a base member, a handle coupled to the base member and configured to be manually contacted at a grip portion of the handle and moved by a hand of a user in one or more degrees of freedom, one or more control input sensors configured to detect positions or orientations of the handle in the one or more degrees of freedom, and a presence sensor coupled to the base member. The presence sensor has a sensing field, and at least a portion of the sensing field is located proximate to the handle.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application claims priority to U.S. Provisional PatentApplication No. 62/912,536, filed Oct. 8, 2019 and titled HAND PRESENCESENSING AT CONTROL INPUT DEVICE, the entire contents of which are herebyincorporated by reference.

BACKGROUND

Control input devices allow a user to control functions of various typesof mechanisms and instruments. Teleoperated surgical devices, forexample, can use various types of medical instruments to performminimally invasive surgical procedures that reduce damage to healthytissue of patients. The medical instruments can be connected tomanipulator devices such as manipulator arms that can be controlled toperform the surgical procedures. Control of the medical instruments at amanipulator device can be provided to an operator at one or more controlinput devices, which may be at a operator terminal or station. Actuatorsof the manipulator device can be controlled by a control input device tocause motion or initiate another function of a medical instrument,camera, or other end effector at the manipulator device that interactswith the patient surgical site. In some examples, the control inputdevice at the operator station can be physically manipulated by theoperator in one or more degrees of freedom to control the end effectorto be moved in coordination with the manipulation of the control device,e.g., to move in corresponding degrees of freedom at the operating site.

In many circumstances, it is desirable for the control system to havethe ability to detect the presence of a user operating the control inputdevices. For example, this allows user control of a manipulator deviceto be enabled when the user is detected to be operating the controlinput device, and safely disabled when the user is not detected to bepresent. In some control systems, the user's presence can be detectedusing one or more presence sensors. For example, some systems include anoperator terminal at which control input devices are used and whichincludes a video output device. An optical detector can detect thepresence of an operator's head when the head is positioned to view thevideo output device. However, such detection does not directly indicatewhether the user's hands are ready to use the control input devices.Furthermore, some control input devices have a structure and/or gripsfor the user's hands that may cause difficulty in directly sensingpresence of user's hands operating the control input device. Forexample, rotating pincher grips on a control input device can provide apinching motion, and they may rotate about one or more axes of thecontrol device, which may cause an operator's hand to adopt a variety ofconfigurations. In some cases, the presence of an operating hand in oneor more such configurations may not be easily detected by sensors.

SUMMARY

Implementations of the present application relate to hand presencesensing at a control input device. In some implementations, a controlinput device includes a base member, a handle coupled to the base memberand configured to be manually contacted at a grip portion of the handleand moved by a hand of a user in one or more degrees of freedom, one ormore control input sensors configured to detect positions and/ororientations of the handle in the one or more degrees of freedom, and apresence sensor coupled to the base member. The presence sensor has asensing field, and at least a portion of the sensing field is locatedproximate to the handle.

Various implementations and examples of the control input device aredescribed. For example, in some implementations, the presence sensor isconfigured to detect electromagnetic radiation or an ultrasonic wavethat is directed through space to the presence sensor by a presence ofthe hand in the sensing field of the presence sensor. In someimplementations, the presence sensor is located on a surface of thehandle that is not contacted by the hand during operation of the controlinput device. In some implementations, the portion of the sensing fieldis located in an approach path of the hand when moving toward the handleprior to operating the handle. In some implementations, the handle isexternal to the sensing field.

In some implementations, the sensing field is shaped as, orapproximately as, a cone that increases in width in a direction awayfrom the presence sensor. In some implementations, the sensing field hasa spatial position fixed with respect to a central axis of the handlethat extends between a distal end and a proximal end of the handle. Insome implementations, the handle at least partially extends into thesensing field. In some implementations, the sensing field is located atleast partially in front of an end of the handle.

In some implementations, the presence sensor is a first presence sensor,the sensing field is a first sensing field located at a first side ofthe handle, and the control input device further includes a secondpresence sensor coupled to the base member and configured to detectsecond electromagnetic radiation that is directed through space to thesecond presence sensor by a presence of the hand in a second sensingfield of the second presence sensor. The second sensing field isproximate to the handle and is located at a second side of the handlethat is opposite the first side. For example, the first side can be afirst side of a vertical plane intersecting a central axis of the handleand the second side a second side of the vertical plane.

In some implementations, a signal generated by the presence sensorcomprises a parameter, and the parameter comprises a value thatcorresponds to a variable distance between an object in the sensingfield and the presence sensor. In some implementations, the presencesensor includes an electromagnetic sensor, which includes an emitted anda detector, the emitter configured to emit a first electromagneticsignal in the sensing field and the detector configured to detect thefirst electromagnetic signal reflected from the hand in the sensingfield. In some implementations, the presence sensor includes an opticaltime-of-flight sensor that generates a signal comprising a value thatcorresponds to a variable distance between the hand in the sensing fieldand the presence sensor. In some implementations, the presence sensorincludes a thermopile sensor or thermal imaging camera, that includes adetector configured to detect infrared radiation emitted by the hand inthe sensing field. Other types of sensors can also be used, e.g.,ultrasonic sensor, etc.

In some implementations, a portion of the handle includes a handledistal end, a handle proximal end opposite the handle distal end, and acentral axis defined between the handle distal end and the handleproximal end. The handle distal end is closer than the handle proximalend to the hand. A base portion of the base member includes a basedistal end and a base proximal end opposite the base distal end, thebase portion extending parallel or approximately parallel to the centralaxis of the portion of the handle. The presence sensor is located on thebase distal end that is closer than the base proximal end to the handledistal end.

In some implementations, the handle includes a central portion thatextends along a central axis of the handle between a distal end and aproximal end of the handle, and the handle includes two grip membersextending from the central portion. The two grip members are eachconfigured to be gripped by a corresponding finger of the hand, and thecentral portion is configured to be positioned between at least twofingers of the hand during grip of the handle. The sensing field isconfigured to cover a region including one or more fingers of the handtouching either of the two grip members. In some implementations, theone or more degrees of freedom include a roll degree of freedom, inwhich the handle is rotatable about the central axis of the handle withrespect to the base member in the roll degree of freedom, and thesensing field is configured to include at least a portion of the hand atall orientations of the handle in the roll degree of freedom while thehand grips the handle. In various implementations, the base member isoptionally mechanically grounded or mechanically ungrounded.

In some implementations, a control input device includes a handleconfigured to be manually contacted at a grip portion of the handle andmoved by a hand of a user in one or more degrees of freedom. The handleincludes a central portion that extends along a central axis of thehandle, and the central portion is configured to be positioned betweenat least two fingers of the hand during a grip of the handle by thehand. One or more control input sensors are configured to detectpositions or orientations of the handle in the one or more degrees offreedom, and a presence sensor is coupled to a distal end of the handlethat is proximate to the hand. The presence sensor is configured todetect electromagnetic radiation or an ultrasonic wave that is directedthrough space to the presence sensor by a presence of the hand in asensing field of the presence sensor, and the sensing field is locatedproximate to the handle.

Various implementations and examples of this control input device aredescribed. For example, in some implementations, the handle isconfigured such that a palm of the hand is out of contact with thehandle while the hand grips the grip portion of the handle. In someimplementations, the presence sensor is configured to detect theelectromagnetic radiation or the ultrasonic wave by a presence of thepalm of the hand in the sensing field of the presence sensor. In someimplementations, a signal generated by the presence sensor comprises aparameter, and the parameter comprises a value that corresponds to avariable distance between the detected hand and the presence sensor. Infurther examples, the parameter includes a value that corresponds to adirection of motion of the hand in the sensing field relative to thepresence sensor or a velocity of the hand in the sensing field. In someexamples, the value is provided to a processor and is usable todetermine whether the hand is operating the control input device. Invarious implementations, the sensing field is located in an approachpath of the hand when moving toward the handle prior to operating thehandle, the handle is positioned external to the sensing field, and/orthe sensing field is positioned at least partially in front of an end ofthe handle. In some implementations, the sensing field is shaped as, orapproximately as, a cone that increases in width in a direction awayfrom the presence sensor, and the sensing field has a spatial positionfixed with respect to a central axis of the handle extending between adistal end and a proximal end of the handle. The presence sensor caninclude: an electromagnetic sensor that includes an emitter configuredto emit a first electromagnetic signal in the sensing field and adetector configured to detect the first electromagnetic signal reflectedfrom the hand in the sensing field; a thermopile sensor that includes adetector configured to detect infrared radiation emitted by the hand inthe sensing field; and/or a thermal imaging camera that includes adetector configured to detect the infrared radiation emitted by the handin the sensing field.

In some implementations, a method includes activating a non-controllingmode in which a handle of a control input device is manually moveable bya user in one or more degrees of freedom without moveably controlling amanipulator device that, e.g., corresponds to the control input device,the manipulator device being in communication with the control inputdevice. In the non-controlling mode, a presence of a hand of a userrelative to the handle is sensed in a sensing field of a presencesensor. A portion of the sensing field is located proximate to thehandle. In response to sensing the presence of the hand, a controllingmode of the control input device is activated in which the handle ismoveable by the user in the one or more degrees of freedom to moveablycontrol the manipulator device. In some implementations of the method,the presence sensor is configured to detect electromagnetic radiation oran ultrasonic wave that is directed through space to the presence sensorby a presence of the hand in a sensing field of the presence sensor. Insome implementations, sensing the presence of the hand includes sensingan approach of the hand toward the handle while the hand is in thesensing field prior to contacting and operating the handle.

In some implementations of the method, sensing the approach of the handtoward the handle includes determining a direction of motion of the handrelative to the handle and determining whether the direction of motionis toward the handle. In some implementations, the method furtherincludes determining a velocity of the hand relative to the handle anddetermining that the velocity of the hand meets (e.g., is less than) athreshold velocity, and the activation of the controlling mode isperformed in response to the velocity of the hand meeting the thresholdvelocity. In some implementations, the method further includesactivating the non-controlling mode in response to sensing an indicationthat the hand is no longer operating the handle. In variousimplementations, the indication includes sensing the hand outside athreshold distance from the handle, and/or sensing the hand moving in aparticular direction relative to (e.g., away from) the handle. In someimplementations of the method, activating the controlling mode isperformed only in response to both sensing the presence of the hand andsensing a presence of the user by one or more other presence detectiondevices of a system that includes the control input device, and the oneor more other presence detection devices include a grip sensor of thecontrol input device and/or a head presence sensor of the system.

In some implementations, the method further includes, while in thecontrolling mode, determining a position of the hand relative to areference location of the control input device, and determining, basedon the position of the hand, one or more characteristics of force to beoutput on the control input device, the one or more characteristics offorce including a maximum force magnitude output on the control inputdevice, a gain of force magnitude output on the control input device,and/or a rate at which the force magnitude on the control input deviceis increased. In some implementations, the method further comprises,while in the controlling mode, determining a position of the handrelative to a reference location of the control input device, andadjusting a safety feature of the control input device based on theposition, including: changing parameters used in detection of patternsof motion, acceleration, or direction of the control input device todetect active use of the control input device by the user, and/orphysically limiting a velocity of the control input device in one ormore degrees of freedom by using one or more force output devicescoupled to a mechanism of the control input device.

In some implementations, the method further includes, in the controllingmode, determining a position of the hand relative to a referencelocation of the control input device, and determining detectionparameters of one or more other presence sensors of the control inputdevice based on the position; the other presence sensors are independentand separate from a hand presence sensing system that performs thesensing of the presence of the hand, and the detection parametersinclude a threshold of sensing, a range of sensing, and/or a duration ofsensing. In some implementations, the method further includes, in thecontrolling mode, detecting presence of the user by one or more otherpresence sensors of the control input device, and determining one ormore detection parameters of a hand presence sensing system based on thedetected presence of the user by the one or more other presence sensors,the other presence sensors being independent and separate from the handpresence sensing system that performs the sensing of the presence of thehand, and the one or more detection parameters of the hand presencesensing system including a threshold of sensing, a range of sensing,and/or a duration of sensing.

In some implementations, a method includes activating a controlling modein which a handle of a control input device is manually moveable by auser in one or more degrees of freedom to moveably control a manipulatordevice that is in communication with the control input device. Themethod includes, in the controlling mode, sensing a presence of a handof the user relative to the handle in a sensing field of a presencesensor, a portion of the sensing field positioned proximate to thehandle, and in response to sensing the presence of the hand, activatinga non-controlling mode in which the handle is moveable by the user inthe one or more degrees of freedom without moveably controlling themanipulator device. In some implementations, sensing the change inpresence of the hand includes sensing the hand outside a thresholddistance from the handle, and/or sensing the hand moving in a directionaway from the handle. In some implementations, the method furtherincludes, prior to activating the controlling mode, sensing the presenceof the hand in the sensing field of the presence sensor and sensing apresence of the user by one or more other presence detection devices ofa system that includes the control input device, the other presencedetection devices including a grip of the control input device and/or ahead presence sensor of the system, and activating the controlling modeis performed only in response to sensing the presence of the hand andsensing the presence of the user by the one or more other presencedetection devices.

In some implementations, a control input device includes handle meansfor being manually contacted at a grip portion of the handle means andfor being moved by a hand of a user in one or more degrees of freedom,means for sensing positions or orientations of the handle means in theone or more degrees of freedom, and means for detecting a hand in asensing field proximate to and external to the handle means. In someimplementations, the means for detecting includes means for detectingelectromagnetic radiation or an ultrasonic wave that is directed throughspace to the means for detecting by a presence of the hand in a sensingfield of the means for detecting. In some implementations, the controlinput device further comprises base means for coupling the handle meansto a mechanical ground. In some implementations, the means for detectingis located at a distal end of the handle means that is proximate to thehand. In some implementations, the means for detecting is located at adistal end of a base portion of the base means, and the base portionextends approximately parallel to a central axis of the handle means.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagrammatic illustration of an example implementation of ateleoperated system which can be used with one or more featuresdisclosed herein, according to some implementations;

FIG. 2 is a front elevational view of an example user control system asshown in FIG. 1 , according to some implementations;

FIG. 3 is a perspective view of an example portion of a control inputdevice which can include one or more features described herein,according to some implementations;

FIG. 4 is a perspective view of an example portion of a control inputdevice including an example implementation of a presence sensing system,according to some implementations;

FIGS. 5 and 6 are a top view and a side view, respectively, of theexample control input device of FIG. 4 , according to someimplementations;

FIG. 7 is a side elevational view of the example control input device ofFIG. 4 in which a presence sensor is located at a distal end of ahandle, according to some implementations;

FIG. 8 is a side elevational view of the example portion of a controlinput device of FIG. 4 in which a presence sensor is located at a basemember end of the handle, according to some implementations;

FIG. 9 is a flow diagram illustrating an example method to detect thepresence of a user's hand to operate a control input device, accordingto some implementations;

FIG. 10 is a flow diagram illustrating an example method to determineand/or adjust presence sensing features and/or other system featuresbased on presence sensor data, according to some implementations; and

FIG. 11 is a block diagram of an example control system which can beused in one or more implementations described herein.

DETAILED DESCRIPTION

One or more implementations described herein relate to control inputdevices having a hand presence sensing system. The hand presence sensingsystem is configured to sense the presence of a hand operating a controlinput device and/or a hand positioned near a control input device. Insome implementations, the control input device includes a handleconfigured to be manually contacted at a grip portion of the handle andmoved by a user's hand in one or more degrees of freedom. A presencesensing system includes one or more presence sensors coupled to thehandle and/or to a base member that is coupled to the handle. Eachpresence sensor has a sensing field, and a portion of the sensing fieldis located proximate to the handle. In some examples, detection of theuser's hand in the sensing field, and/or detection of the hand operatingthe handle, causes the system to enter a controlling mode in which thecontrol input device can control functions of a manipulator device.

In various implementations, the presence sensor is configured to detectelectromagnetic radiation or an ultrasonic wave that is directed throughspace to the presence sensor by a presence of the hand in a sensingfield of the sensor. In some examples, the portion of the sensing fieldcan be located in an approach region of the handle, e.g., a region movedinto by a hand that is moving toward the handle prior to operating it.The handle can be external to the sensing field, and/or the handle canextend into the sensing field. The sensing field can be positioned atleast partially in front of an end of the handle. Multiple presencesensors can each provide a sensing field, e.g., on different sides of acentral axis of the handle. The presence sensor can be any of varioustypes, including an electromagnetic sensor (e.g., a time-of-flightsensor), a thermal sensor (e.g., a thermopile sensor or thermal imagingcamera operative to detect infrared radiation emitted by the hand in thesensing field), an ultrasonic sensor, etc.

Features described herein provide a control input device with severaladvantages. In some prior systems, the lack of a positive indicationthat a user's hand is controlling a user input device may result in asituation in which a manipulator device moves as a result of acorresponding control input device movement, but the control inputdevice movement was unplanned. For example, to avoid unintendedmanipulation of a manipulator device even if a user is viewing a displayscreen on which an image of the manipulator device is shown, controlover the manipulator device by a control input device can be enabledonly if the user's hand is positioned to control the control inputdevice properly. Also, if motors on the control input device are used toprovide haptic feedback for a user, a situation may exist in which thehaptic feedback could push the control input device away from the user'shand if the user's hand does not have a proper grip on the control inputdevice. Features described herein provide robust detection of a user'shand on or near the control input device, thus allowing a controlledmanipulator system to enter a controlling mode even more safely than insystems providing no such hand presence detection.

Furthermore, in some implementations, the described control input devicecan provide detection of the user's hand within a proximity of thecontrol input device and without contact of any portion of the hand tothe surface of the control device by using non-contact sensors. Suchnon-contact sensors can more reliably sense hand presence than manytypes of contact sensors. For example, such proximity detection allowsmore robust hand detection for some types of control input devices thatare not operated using direct contact with the palm of a hand, butrather with a user's fingertips or other hand portion (e.g., if the handencloses a spatial region around the control input device). Describedfeatures allow the presence of the hand to be reliably detected whenoperating such a control input device. Features also allow a system touse the described presence sensing system in conjunction with otherpresence sensing systems (e.g., sensing of a user's head or other userbody portions) to provide more robust user presence detection. This useof multiple sensing features can allow easier and faster detection ofuser presence in a position to properly operate an input device.

Furthermore, proximity detection allows the system to more robustlydetermine to enter or exit a controlling mode. For example, detection ofthe user's hand in the proximity of the control input device alerts thesystem of the user's intent to grip the control input device handle. Insome implementations, the direction of hand movement can be detected,and this detected direction of hand movement allows further detectionand determination of user intent. If the user's hand is not in a definedproximity of the control input device, or when the hand is detected tomove in a direction relative to the control input device (e.g., in adirection away from the control input device), the detection of handproximity or direction of hand movement can be used to either exit thecontrolling mode or to not enter the controlling mode, as the case maybe.

Thus, features of the presence sensing system can determine if handmovement near the control input device may be accidental orunintentional based on hand motion. For example, accidental motion ofthe control input device (e.g., because the control input device wasbumped by an object other than the user's hand) can be detected andignored if the user's hand is not detected near to the control inputdevice. Furthermore, system software can use hand detection informationto make decisions about system operating state and to inform safetyalgorithms that can trigger actions (e.g., system operating statechanges) when necessary. Detected user intent based on hand motion canbe used to provide features for displayed user interfaces, otherfunctions of the system, safety features, power-saving features, etc.For example, a user interface (and/or other system components) can beturned on from an unpowered or low-power state if user intent to use thecontrol input device is detected.

Various implementations described herein are compact, robust, andinexpensive. Using various described features, determination by a systemto enter and exit controlling mode is made more easily, reliably, androbustly.

The terms “center,” “parallel,” “perpendicular,” “aligned,” orparticular measurements in degrees, Hertz, or other units as used hereinneed not be exact and can include typical engineering tolerances. Someimplementations herein may relate to various objects in terms of theirstate in three-dimensional space. As used herein, the term “position”refers to the location of an object or a portion of an object in a threedimensional space (e.g., three degrees of translational freedom alongCartesian X, Y, Z coordinates). As used herein, the term “orientation”refers to the rotational placement of an object or a portion of anobject (three degrees of rotational freedom—e.g., roll, pitch, and yawaround the Cartesian X, Y, and Z axes). As used herein, the term “pose”refers to the position of an object or a portion of an object in atleast one degree of translational freedom and to the orientation of thatobject or portion of the object in at least one degree of rotationalfreedom (up to six total degrees of freedom).

As referred to herein, a mechanically grounded unit or device isconstrained with respect to possible position and orientation motion ina large working environment (e.g., an operating area or room). Also,such a unit is kinematically coupled to the ground (e.g., mechanicallysupported by a console, supports, or other object attached to theground). As used herein, the term “proximal” refers to an element thatis close to (or closer to) a mechanical ground and the term “distal”refers to an element that is away from (or further from) a mechanicalground.

The term “finger,” as used herein, refers to any digit of the hand,e.g., thumb, index finger, middle finger, ring finger, or pinky finger.

FIG. 1 is a diagrammatic illustration of an example teleoperatedsurgical system 100 which can be used with one or more featuresdisclosed herein. Other types of control systems or other systems can beused in other implementations involving described features. Teleoperatedsurgical system 100 includes a user control system (e.g., surgeon'sconsole) 102 and a manipulator system 104.

In this example, the user control system (e.g., surgeon's console) 102includes a viewer 213 (shown in FIG. 2 ) where an image of a worksite isdisplayed during an operating procedure using the system 100. Forexample, the image can be displayed by a display device, such as one ormore display screens, to depict a surgical site during a surgicalprocedure. A support 110 is provided on which a user 112, e.g., anoperator such as a surgeon, can rest forearms while gripping two controlinput devices 210 and 212 (shown in FIG. 2 ), one in each hand. Thecontrol input devices can be positioned in a workspace 114 disposedinwardly beyond the support 110. When using the user control system 102,the user 112 can sit in a chair in front of the control system 102,position the user's head/eyes in front of the viewer, and grip thecontrol input devices 210 and 212, one in each hand, while restingforearms on the support 110. Additional example details are describedbelow with reference to FIG. 2 .

The teleoperated system 100 may also include a manipulator system 104which can be controlled by the user control system 102. For example,manipulator system 104 can be or include a manipulator device. In someimplementations as shown, during a surgical procedure, the manipulatorsystem 104 can be positioned close to a patient on an operating tableworksite for surgery (or close to other to other type of worksite),where it can remain stationary until a particular surgical procedure orstage of a procedure is completed.

Manipulator system 104 can include one or more manipulator armassemblies 120. In some examples, an arm assembly 120 can includemultiple links rotatably coupled to each other. Portions of the armassembly can be actuated with a motor and sensed about rotational axes.In some examples, one or more of the arm assemblies 120 can beconfigured to hold an image capturing device, e.g., an endoscope 122,which can provide captured images of a portion of the surgical site. Insome implementations, the captured images can be transmitted to theviewer of the user control system 102 and/or transmitted to one or moreother displays, e.g., a display 124 coupled to the manipulator system104.

In some examples, one or more of the arm assemblies 120 may each includea surgical instrument 126. Each surgical instrument 126 can include asurgical end effector, e.g., for treating tissue of the patient. An endeffector can be provided the degrees of freedom provided by, e.g., therotation of link members of the associated arm assembly, linear motionby an end effector mechanism, etc. Components in the arm assembly canfunction as force transmission mechanisms to receive teleoperated servoactuation forces and redirect the received forces to operate componentsof the end effector. An end effector can include one or more motors orother actuators that operate associated features of the end effector,such as the pitch, yaw, and/or roll of the end effector, opening jaws ormoving a blade of the end effector, the output of material transportedthrough a connecting tube (e.g., liquid or other fluids), suctionforces, and/or any of a multiple of other end effector functions. Endeffector mechanisms can include flexible elements, articulated “snake”arms, steerable guide tubes, catheters, scalpel or cutting blade,electro-surgical elements (e.g., monopolar or bipolar electricalinstruments), harmonic cutter, scissors, forceps, retractors, dilators,clamps, cauterizing tools, needles, needle drivers, staplers, drills,probes, scopes, light sources, guides, measurement devices, vesselsealers, laparoscopic tools, or other tip, mechanism or device. Oneexample of a surgical manipulator arm is a da Vinci® surgical systeminstrument manipulator arm in surgical systems commercialized byIntuitive Surgical, Inc. of Sunnyvale, Calif.

In this example, the arm assemblies 120 can be caused to move andarticulate the surgical instruments 126 in response to manipulation ofcorresponding control input devices, e.g., manipulation of the controlinput devices 210 and 212 (shown in FIG. 2 ) at the user control system102 by the user 112. This arrangement allows user 112 to direct surgicalprocedures at internal surgical sites through minimally invasivesurgical apertures. For example, one or more actuators coupled to thearm assemblies 120 can output force to cause links or other portions ofthe arm assemblies to move in particular degrees of freedom in responseto control signals received from the user control system 102. Forexample, movement of an arm and end effector in one or more degrees offreedom can correspond to movement in one or more degrees of freedom ofan associated control input device handle by a user. The user controlsystem 102 can be used within a room (e.g., an operating room) with themanipulator system 104 or can be positioned more remotely from themanipulator system 102, e.g., at a different location than themanipulator system.

Some implementations of the teleoperated system 100 can providedifferent modes of operation. In some examples, in a non-controllingmode (e.g., safe mode) of the teleoperated system 100, the controlledmotion of the manipulator system 104 is disconnected from the controlinput devices of the user control system 102 in disconnectedconfiguration, such that movement and other manipulation of the controlinput devices does not cause motion of the manipulator system 104. In acontrolling mode of the teleoperated system (e.g., following mode, inwhich one or more manipulator instruments or other devices follow acorresponding control input device), motion of the manipulator system104 can be controlled by the control input devices 210 and 212 of theuser control system 102 such that movement and other manipulation of thecontrol input devices causes motion of the manipulator system 104, e.g.,during a surgical procedure.

Some implementations can be or include a teleoperated medical systemsuch as a da Vinci® Surgical System (e.g., a Model IS3000 or IS4000,marketed as the da Vinci Si® or da Vinci Xi® Surgical System),commercialized by Intuitive Surgical, Inc. of Sunnyvale, Calif. However,features disclosed herein may be implemented in various ways, includingin implementations at least partially computer-controlled, controlledvia electronic control signals, manually controlled via direct physicalmanipulation, etc. Implementations on da Vinci® Surgical Systems aremerely exemplary and are not to be considered as limiting the scope ofthe features disclosed herein. For example, different types ofteleoperated systems having manipulator devices at worksites can makeuse of actuated controlled features described herein. Other,non-teleoperated systems can also use one or more described features,e.g., various types of control systems and devices, peripherals, etc.

In some implementations, a controlled manipulator device can be avirtual representation of device, e.g., presented in a graphicalsimulation provided by a computing device coupled to the teleoperatedsystem 100. For example, a user can manipulate the control input devices210 and 212 of the user control system 102 to control a displayedrepresentation of an end effector in virtual space of the simulation,similarly as if the end effector were a physical object coupled to aphysical manipulator device.

FIG. 2 is a front elevational view of an example user control system 102as described above for FIG. 1 . User control system 102 includes aviewer 213, where an image of a worksite can be displayed during aprocedure using the teleoperated system 100. For example, imagesdepicting a surgical site can be displayed during a surgical procedure.The viewer 213 can be positioned within a viewing recess 211 in whichthe user can position his or her head to view images displayed by theviewer 213. When using the user control system 102, the user 112 can sitin a chair in front of the user control system and position his or herhead within the recess 211 such that his or her eyes are positioned infront of the viewer 213.

In some implementations, one or more user presence sensors 214 can bepositioned at one or more locations of the user control system 102 todetect the presence of a user located next to or near to the usercontrol system 102. In this example, the user presence sensors 214 cansense a presence of a user's head within the recess 211. For example, anelectromagnetic sensor (e.g., optical sensor) can be used for a presencesensor. In some examples, the optical sensor can include an emitter 216and a detector 218. A beam of infrared or other wavelength of light isemitted from one side of the recess 211 by the emitter 216, and the beamis detected on the other side of the recess by the detector 218. If thebeam is interrupted from detection by the detector, e.g., due to theuser's head blocking the beam, then the system determines that a user'shead is within the recess and that the user is in a proper position touse the control input devices of the user control system 102. Additionalor alternative types of presence sensors can be used in variousimplementations.

Two control input devices 210 and 212 are provided for usermanipulation. In some implementations, each control input device 210 and212 can be configured to control motion and functions an associated armassembly 120 of the manipulator system 104. For example, a control inputdevice 210 or 212 can be moved in a plurality of degrees of freedom tomove a corresponding end effector of the manipulator system 104 incorresponding degrees of freedom. In some implementations, the controlinput devices are manual input devices which can be moved in all sixCartesian degrees of freedom.

The control input devices 210 and 212 are positioned in workspace 114inwardly beyond the support 110. For example, a user 112 can restforearms while gripping the two control input devices 210, 212, with onecontrol input device in each hand. The user also positions his or herhead within the viewing recess 211 to view the viewer 213 as describedabove while manipulating the control input devices 210 and 212. Variousexamples of portions of input devices that can be used as control inputdevices 210 and 212 are described below.

Some implementations of user control system 102 can include one or morefoot controls 220 positioned below the control input devices 210 and212. The foot controls 220 can be depressed, slid, and/or otherwisemanipulated by a user's feet to input various commands to theteleoperated system while the user is sitting at the user control system102.

FIG. 3 is a perspective view of an example controller portion 300 of acontrol input device which can include one or more features describedherein. In some implementations, the control input device can be part ofa system in which user input provided via the control input device isused to control one or more controllable device functions. For example,the system can be a teleoperated system in which the control inputdevice is, or is included in, a master device that controls amanipulator device (e.g., slave device). For example, controller portion300 can be used as a portion of an input control device that is acontrol input device 210 or 212 as described above with reference toFIGS. 1 and 2 , or portion 300 can be included in a different controldevice. In some implementations, the controller portion 300 includes oneor more gimbal mechanisms.

Controller portion 300 includes a handle 302 which is contacted by auser to manipulate the control input device. In this example, the handle302 includes two grips that each include a finger loop 304 and a gripmember 306 (grip members 306 a and 306 b). The two grip members 306 arepositioned on opposite sides of a central portion 303 of the handle 302,and the grip members 306 can be grasped, held, or otherwise contacted bya user's fingers. Each finger loop 304 is attached to a respective gripmember 306 and can be used to secure a user's fingers to the associatedgrip member 306. In this example, finger contacts 305 can be connectedor formed at the unconnected end of the grip members 306 a and 306 b toprovide surfaces to contact the user's fingers. The user may alsocontact other portions of handle 302 while grasping the grip members306.

Each grip member 306 and finger loop 304 can be moved in an associateddegree of freedom 308 (e.g., 308 a and 308 b). In some examples, thegrip members 306 a and 306 b are each coupled to the central portion 303of the handle 302 at respective rotational couplings, allowingrotational movement of the grip members about grip axes 307 a and 307 b,respectively, with respect to the central portion 303. Each grip member306 a and 306 b can be moved in an associated degree of freedom 308 aabout axis 307 a and degree of freedom 308 b about axis 307 b,respectively, e.g., by a user contacting the grip members. For example,in some implementations the grip members 306 a and 306 b can be movedsimultaneously in a pincher-type of movement (e.g., toward or away fromeach other). In various implementations, a single grip member 306 andfinger loop 304 can be provided, or only one of the grip members 306 canbe moved in the degree of freedom 308 while the other grip member 306can be fixed with reference to the handle 302. For example, thepositions of grip members 306 a and 306 b in their degrees of freedomcan control corresponding rotational positions of an end effector orcomponent thereof.

One or more grip sensors (not shown) can be coupled to the handle 302and/or other components of the controller portion 300 and can detect thepositions of the grip members 306 a and 306 b in their degrees offreedom 308. The grip sensors can send signals describing sensedpositions and/or motions to one or more control circuits of theteleoperated system 100. In some modes or implementations, the controlcircuits can provide control signals to a manipulator device, e.g.,manipulator system 104. For example, the positions of the grip members306 a and 306 b in degrees of freedom 308 a and 308 b can be used tocontrol any of various degrees of freedom of an end effector of themanipulator system 104, some examples of which are described herein.

Various implementations of the controller 300 can provide one or moreactive actuators (e.g., motors, voice coils, etc.) to output activeforces on the grip members 306 in the degrees of freedom 308. Forexample, a sensor and/or actuator can be housed in central portion 303or in housing 309 and coupled to the grip members 306 by a transmission.Some implementations can provide one or more passive actuators (e.g.,brakes) or springs between the grip members 306 and the central portion303 of the handle 302 to provide resistance in particular directions ofthe grips (e.g., movement in directions toward each other in degree offreedom 308).

Handle 302 is additionally provided with a rotational degree of freedom310 about a roll axis 312 defined between a first end and second end ofthe handle 302. The roll axis 312 is a longitudinal axis in this examplethat extends approximately along the center of the central portion 303of handle 302. Handle 302 can be rotated about axis 312 with respect toa base member of the controller portion 300, such as a base member thatincludes housing 309. For example, a user can rotate the grip members306 and central portion 303 as a single unit around the axis 312, withrespect to housing 309, to provide control of a manipulator device, suchas an end effector of the manipulator system 104 or other element of themanipulator system.

One or more control input sensors (not shown) can be coupled to thehandle 302 to detect the orientation of the handle 302 in the rotationaldegree of freedom 310. For example, the sensor can send signalsdescribing the orientation to control circuits of the teleoperatedsystem 100 which can provide control signals to the manipulator system104 similarly as described above. For example, rotation of handle 302 indegree of freedom 310 can control a particular degree of freedom of anend effector of the manipulator system 104 that is different than amanipulator degree of freedom controlled by degree of freedom 308 of thegrip members 306.

Some implementations of the controller portion 300 can provide one ormore actuators to output forces on the handle 302 (including gripmembers 306 and finger loops 304) in the rotational degree of freedom310. For example, a sensor and/or actuator can be housed in housing 309and coupled to the handle 302 by a shaft extending through the centralportion 303 of the handle 302.

In various implementations, the handle 302 can be provided withadditional degrees of freedom. For example, a rotational degree offreedom 320 about a yaw axis 322 can be provided to the handle 302 at arotational coupling between an elbow shaped link 324 and a link 326,where the elbow shaped link 324 is coupled to the handle 302 (e.g., athousing 309). In this example, yaw axis 322 intersects and is orthogonalto the roll axis 312. For example, yaw axis 322 can be similar to axis232 shown in FIG. 2 . Additional degrees of freedom can similarly beprovided. For example, link 326 can be elbow-shaped and a rotationalcoupling can be provided between the other end of link 326 and anotherlink (not shown). A rotational degree of freedom 328 about an axis 330can be provided to the handle 302 at the rotational coupling. Forexample, axis 330 can be similar to axis 230 shown in FIG. 2 . In someexamples, the controller portion 300 can allow movement of the handle302 within the workspace 114 of the user control system 102 with aplurality of degrees of freedom, e.g., six degrees of freedom includingthree rotational degrees of freedom and three translational degrees offreedom. One or more additional degrees of freedom can be sensed byassociated control input sensors and/or actuated by actuators (motors,etc.) similarly as described above for the degrees of freedom 308 and310, the sensors and actuators coupled to portion 300. In variousimplementations, sensors can sense positions of the handle in a degreeof freedom, or sense orientations of the handle in a degree of freedom,or sense positions and orientations of the handle in multiple degrees offreedom. For example, positions in a translational degree of freedom andorientations in a rotational degree of freedom can be sensed by one ormore control input sensors associated control input sensors. In someexamples, a position in a translational degree of freedom and/ororientation in a rotational degree of freedom can be derived fromrotations of components (e.g., links of a linkage) coupled to the handle302 as sensed by rotational sensors. Some implementations can includelinear sensors that can directly sense translational motion of one ormore components coupled to the handle 302. In some implementations, eachadditional degree of freedom of the handle 302 can control a differentmanipulator degree of freedom (or other motion) of an end effector ofthe manipulator system 104.

In an example implementation, handle 302 is mechanically grounded, i.e.,supported in space by a kinematic chain with an end stationary atmechanical ground, such as a floor, wall, or ceiling. For example, thehousing 309 can be coupled to a mechanical linkage that is coupled tothe ground or an object connected to ground, providing a stable platformfor the use of the hand controller portion 300. For example, a groundedmechanical linkage can be connected to the base member, e.g., with oneor more rotary couplings, ball joints, or other couplings, includinglinear joints. The mechanical linkage can provide six or more degrees offreedom to the handle 302. In some implementations, one or more links inthe linkage can include links 324 and 326.

In some examples, the base member can be coupled to a serial kinematicchain, the proximal end of which is mechanically grounded. The kinematicchain can include multiple members or links that are rotatably coupledto one or more other members or links of the chain, e.g., by rotationalor linear couplings. The rotational axes of the chain can be sensedand/or driven by sensors and/or actuators. Some implementations canprovide additional actuated and/or sensed motion of the kinematic chain,e.g., about axes extending lengthwise through one or more members. Insome implementations, multiple members of the kinematic chain form agimbal mechanism that allows the handle 302 to be rotated about therotational axes of the chain. In some implementations, the handle 302can also be translated in at least three linear degrees of freedomallowed by the kinematic chain.

Various kinematic chains, linkages, gimbal mechanisms, flexiblestructures, or combinations of two or more of these can be used with themechanically grounded hand controller in various implementations toprovide one or more degrees of freedom to the hand controller. Someexamples of such implementations are described in U.S. Pat. No.6,714,839 B2, incorporated herein by reference.

In the described example, handle 302 includes one or more controlswitches 350, e.g., coupled to the central portion 303 or to mechanismswithin central portion 303. For example, two control switches 350 can bepositioned on opposite sides of axis 312, and/or additional controlswitches can be provided. In some examples, a control switch 350 has aportion that can slide parallel to the axis 312, e.g., as directed by auser's finger, or the control switch portion can be depressed. In someimplementations, the control switch 350 can be moved to variouspositions to provide particular command signals, e.g., to selectfunctions, options, or modes of the control console and/or control inputdevice (e.g., a controlling mode or non-controlling mode as describedherein), to command a slave device or other system in communication withthe control input device, etc. In some implementations, one or more ofthe control switches 350 can be implemented as a button (e.g., depressedin a direction, such as perpendicular to the axis 312 or otherdirection), a rotary dial, a switch that moves perpendicular to the axis312, or other type of input control. Control switch 350 can useelectromagnetic sensors, mechanical switches, magnetic sensors, or othertypes of sensors to detect positions of the switch.

Handle 302 also includes a hand presence sensing system including one ormore presence sensors that can detect the presence of a user's handoperating the handle, detect the user's hand approaching or leaving thehandle, detect a hand approaching or leaving the handle as well as apresence of the user's hand operating the handle, etc. Variousimplementations of presence sensors are described below with respect toFIGS. 4-8 .

One or more features described herein can be used with other types ofcontrol input devices. For example, controller portion 300 can be or bea portion of a mechanically ungrounded control input device which isfree to move in space and is disconnected from ground. As used herein, amechanically ungrounded control input device refers to a control inputdevice that is unconstrained with respect to possible position andorientation motion in a working environment (e.g., an operating area orroom). Also, such a control device is kinematically separated from theground, e.g., not mechanically supported by a console, supports, orother object attached to the ground. In some implementations, amechanically ungrounded control device may be in tethered or untetheredconnection with one or more associated components such as controlprocessors, data sources, sensors, power supplies, etc. For example, thecontrol device may be tethered, e.g., connected physically to thesecomponents via a cable or wire, or untethered, e.g., not physicallyconnected to such components and in communication with the componentsvia wireless communication signals.

In some examples, one or more handles similar to handle 302 and/or gripmembers 306 can be coupled to a mechanism worn on a user's hand andwhich is ungrounded, allowing the user to move grips freely in space. Insome examples, the positions of the grips relative to each other and/orto other portions of the handle can be sensed by a mechanism couplingthe grips together and constraining their motion relative to each other.Some implementations can use glove structures worn by a user's hand.Furthermore, some implementations can use sensors coupled to otherstructures to sense the grips within space, e.g., using video cameras orother sensors that can detect motion in 3D space. Some examples ofungrounded control input devices are described in U.S. Pat. No.8,543,240 B2 (filed Sep. 21, 2010) and U.S. Pat. No. 8,521,331 B2 (filedNov. 13, 2008), both incorporated herein by reference in theirentireties.

FIG. 4 is a perspective view of an example implementation of acontroller portion 400 of a control input device including an exampleimplementation of a presence sensing system. FIG. 5 is a top plan viewof the controller portion 400, and FIG. 6 is a side elevational view ofthe controller portion 400.

In some implementations, the controller portion 400 can be implementedas the control portion 300 described above with respect to FIG. 3 , orcan be included in a different input control device.

Controller portion 400 includes a handle 402 (shown in cross section)coupled to a base member 408, which can be similar to handle 302 andhousing 309 as described for FIG. 3 .

Handle 402 includes a first end (proximal end) 404, a second end (distalend) 405 opposite the first end, and a central axis 412 defined betweenthe first and second ends. A central portion 407 can extend between theproximal end 404 and distal end 405. Handle 402 (e.g., a roll member)can be rotated about central axis 412 in a roll degree of freedom 413with respect to the base member 408. In some implementations, handle 402can include the grip members 406 that are rotationally coupled to acentral portion 407 that extends along the central axis 412, similarlyas grip members 306 of FIG. 3 . Central portion 407 is configured to bepositioned between at least two fingers of a hand during grip of thehandle by the hand, similarly as described for FIG. 3 . One or morecontrol input sensors (e.g., roll sensors) can be coupled to thecontroller portion 400 and detect the roll (rotary) orientation ofhandle 402 about axis 412. The roll sensors can send signals describingsensed orientations and/or motion to a control circuit of theteleoperated system 100. In some modes or implementations, the controlcircuit can provide control signals to the manipulator system 104. Insome implementations, an actuator (e.g., motor) can be used to driverotation of handle 402 about central axis 412.

Base member 408 is rotationally coupled to handle 402, allowing handle402 to rotate about axis 412 with respect to the base member 408. Basemember 408 can have a variety of shapes and can include portions orextensions in various configurations. In an example implementation, basemember 408 is mechanically coupled to a ground such that handle 402 ismechanically grounded, e.g., via one or more links (such as links 324and 326 as described above). In other implementations, base member 408is mechanically ungrounded.

In the example of FIG. 4 , base member 408 includes a first base portion420, a second base portion 421, and a third base portion 422. First baseportion 420 is rotatably coupled to handle 402. Second base portion 421extends from the first base portion 420. In various implementations,second base portion 421 extends approximately orthogonally to thecentral axis 412 of handle 402 as shown, or can extend at other anglesrelative to the central axis 412. Third base portion 422 extends fromthe second base portion. In various implementations, third base portion422 cab extend approximately parallel to a central axis 412 of handle402, or can extend at other angles relative to the central axis 412. Forexample, a distal end (e.g., portion) 424 of parallel portion 422 can berotationally coupled to another base member, similarly as described forFIG. 3 .

Controller portion 400 includes a presence sensing system 430 that iscoupled to the parallel portion 422 of the base member 408. Presencesensing system 430 includes one or more presence sensors that senseobjects in one or more sensing fields in space. Herein, a “sensingfield” can include multiple individual sensing fields, e.g., eachindividual sensing field provided by a corresponding one of multiplesensors. In some implementations, sensor(s) of the presence sensingsystem 430 detect a presence of an object in the sensing field. Forexample, the sensor can detect electromagnetic radiation (or ultrasonicwave, as described below) that is directed through space to the sensorby a presence of an object in the sensing field of the sensor, such as ahand. In response to detecting the object, the presence sensor generatesone or more signals that are sent to a control circuit for the controlinput device. For example, in some implementations the signal caninclude a parameter, e.g., a value that indicates the detection of anobject and/or corresponds to a variable distance between the object(e.g., hand) and the presence sensor (or other reference location). Theparameter can also or alternatively indicate other characteristics,e.g., velocity of the object.

In the example of FIG. 4 , presence sensors 440 and 442 (describedbelow) are used and are positioned at the distal end (e.g., portion) 424of the parallel portion 422 that extends parallel to the central axis412. For example, the distal end 424 can be closer than the proximal end(e.g., portion) 425 of the parallel portion 422 to the distal end 405.In some implementations, the presence sensing system 430 can include oneor more optional sensors at one or more other locations of thecontroller portion 400. In some examples, one or more optional sensors450 are positioned at the distal end 405 of handle 402, as describedbelow with respect to FIG. 7 . In some implementations, one or moresensors 460 are positioned on the base portion 420 of the base member408, as described below with respect to FIG. 8 .

In some implementations, the presence sensor(s) use a direct linear viewthrough space to detect a hand, such that at least a portion of the handshould be unobstructed in a linear view path to the sensor to allowdetection. The presence sensors are placed such that components ofhandle 402 and controller portion 400 do not obstruct the sensing field.Thus, the example locations of placement for sensors 440, 442, 450, and460 provide unobstructed sensing fields to detect at least a portion ofa hand during its operation of the control input device, and/or todetect a hand near handle 402. In this example, the presence sensors arealso placed so that a hand must be positioned within the sensing fieldas the hand approaches the handle (for detecting hand approach orproximity) and when the hand is in a position to operate the handle (fordetecting hand presence in an operating position).

In some implementations, as shown in the example of FIG. 4 , sensors 440and 442 provide two individual sensing fields 432 and 434 (e.g., fieldsof view), respectively, directed to different regions of space nearhandle 402. In this example, sensing field 432 is directed to a regionof space that is on a first side of the central axis 412 (e.g., on afirst side of central portion 407 and grip members 406 of handle 402),and sensing field 434 is directed to a different region of space that ison a second side of the central axis 412 (e.g., on a second side ofcentral portion 407 and grip members 406). For example, the differentsides can be different sides of a vertical plane that extends throughthe central axis 412. For example, the vertical plane can be oriented inthe vertical direction with reference to FIG. 4 or 6 . In someimplementations, the first side can be a left side of a vertical planethat extends through the central axis 412 and central portion 407, andthe second side can be a right side of the vertical plane that extendsthrough the central axis 412 and central portion 407, with respect to auser positioned on the side of distal end 405.

An example of sensing fields 432 and 434 positioned on the left andright sides, respectively, of central axis 412 as described above, whichare provided by sensors 440 and 442 positioned on the parallel portion422, is shown in FIG. 5 . In this example, the sensing fields do notoverlap, and the central axis 412 extends between the sensing fields 432and 434 without entering or intersecting either of the sensing fields.

Additionally or alternatively, the sensing fields 432 and 434 can bepositioned at least partially in front of handle 402 from a user'sperspective, e.g., between distal end 405 of handle 402 and a user. Forexample, as shown in FIG. 5 , the sensing fields 432 and 434 arepositioned to the sides of the central axis 412 and are partiallypositioned in front of the distal end 405 such that center axes 436 and438 of the sensing fields 432 and 434, respectively, are in front of thedistal end 405 of handle 402.

In some implementations, as shown in FIG. 6 , the sensing fields 432 and434 are at least partially positioned in a spatial region extending pastan end of handle 402. For example, the sensing fields 432 and 434 can bepositioned at least partially in front of the distal end 405 of handle402 (e.g., to either side or both sides of axis 412 as shown in FIG. 5), with respect to a user.

In some implementations, sensor 440 can provide sensing field 432 andsensor 442 can provide sensing field 434. In some implementations, asingle sensor can provide multiple individual sensing fields, e.g.,sensing fields 432 and 434 and/or additional sensing fields. In someimplementations, a sensing field can be a combination of multipleindividual sensing fields. In the implementation shown, the sensors ofthe presence sensing system, including sensors 432 and 434, are notlocated on a surface of the handle that is contacted by the hand duringoperation of the control input device, nor do they sense such contact ofthe user with such contacted surfaces of the handle.

Fingers of a hand operating handle 402 may contact grip members 406 asshown, such that the operating hand is present in at least one of thesensing fields 432 and 434. For example, the sensing field is positionedsuch that the hand is included in the sensing field in response to oneor more fingers of the hand touching either of the two grip members.

The sensing field(s) are configured to include at least a portion of thehand in response to one or more fingers of the hand touching either ofthe two grip members 406. Thus, the position of the sensing fields 432and 434 on the sides of central axis 412 in the example of FIGS. 4-6allows these fields to sense portions of the user's hand while the handoperates the control input device. In some implementations, handportions closer to the wrist can be sensed (e.g., a side of the palmportion of the hand) if fingers of the hand (or a portion of one of morefingers) are outside the sensing fields 432 and 434 during operation.

In some implementations, the left and right placement of the sensors canprovide more robust sensing than a sensor that is centered to pointdirectly at the distal end 405 or directly in front of distal end 405(e.g., intersecting axis 412). For example, a disadvantage of someimplementations of centered sensors is that it may be possible, in somehand grip configurations, for a centered sensing field to be in a gapbetween the users fingers and miss detection of the hand. The left andright placement allows the sensors to detect the regions to the sides ofthe grip mechanisms 406, e.g., without detecting the distal end 405. Insome implementations, a single sensor pointed towards the distal end 405can be used, e.g., if the sensing field of the sensor is sufficientlywide to detect the hand in various possible hand grip configurations.

In some implementations, as shown in FIGS. 4-6 , at least a portion ofthe sensing field is located in an approach region or path of a handwhen the hand moves toward the handle prior to operating the handle. Forexample, the hand enters one or more of the sensing fields as the handapproaches the handle with user intent to operate the handle. In someimplementations, the sensing field has an orientation and/or size suchthat an object, such as a hand, can be sensed within as well as outsidea particular designated region, e.g., sensed within or greater than adesignated threshold distance as described herein.

The orientation, size, and/or shape of sensing fields 432 and 434 can bebased on the type of sensors 440 and 442 that are used to detect apresence of a hand of a user. Some examples of types of sensors whichcan be used for sensors 440 and 442 are described below.

In some implementations, each sensing field 432 and 434 can be shaped asa cone. For example, the sensing field 432 can have a particular widthat the sensor 440 and increases in width in a direction away from thesensor 440, and a similar sensing field 434 can be provided by sensor442. Herein, the term “cone” or “conical” refers to an approximate coneshape, which does not necessitate an exact conical geometry, e.g.,manufacturing tolerances, interference patterns, warps due toobstructions such as handle 402, or other allowances can be included inthe conical sensing field. Furthermore, this term can refer to coneshaving circular cross sections, as well as or alternatively crosssections of other shapes, e.g., ellipses, ovals, rectangles, squares,triangles, etc. In some implementations, each sensing field 432 and 434can be shaped as a cylinder, rectangle, or other shape. Each cone has adepth and volume limited by a sensing range of the associated sensor 440or 442. In some implementations, the sensing field shape can be madewider or narrower, e.g., as appropriate to cover regions that areproximate to and/or intersected by the distal end 405 of handle 402. Insome implementations, the sensing field can be limited to a particularsize, e.g., depth and/or volume, that may be less than the sensorcapability of the sensor. For example, the depth can be limited to aparticular distance from the sensor at which the sensor can detectobjects in its sensing field. In some examples, the sensing field can belimited, e.g., in depth and/or volume, so that other portions orcomponents of the control input device (or components of a systemincluding the control input device) are not potentially erroneouslydetected as hands.

In some implementations, the sensing fields 432 and 434 can partiallyoverlap. For example, in an alternative implementation of FIG. 5 , thesensing fields 432 and 434 can overlap in front of the distal end 405 ofhandle 402 such that central axis 412 extends through both sensingfields 432 and 434.

In some implementations, handle 402 is proximate to and external to(e.g., outside) the sensing fields 432 and 434. In some examples, handle402 is not present in and does not extend into the sensing fields. Withsome types of sensors that detect electromagnetic radiation signalsreflected from an object in the sensing field, the handle being externalto the sensing fields allows only new objects present in the sensingfield to reflect the signals.

In some implementations, a portion of handle 402 can extend into one ormore of the sensing fields of the presence sensing system, e.g., intosensing fields 432 and 434 such that handle 402 intersects one or moreof the sensing fields. For example, the distal end 405 of handle 402 canextend into one or both sensing fields 432 and 434. With some types ofsensors, reflected signals caused by the components of handle 402 can benormalized such that such handle components 402 are ignored and newobjects located within the sensing field are detected by the time offlight sensors.

The sensing fields 432 and 434 have spatial positions that are fixedwith respect to the central axis 412 of handle 402, e.g., fixed withrespect to the spatial position of handle 402. In some examples, sensors440 and 442 that emit the sensing fields 432 and 434 are positioned onthe parallel portion 422 of the base member 408 that is fixed withrespect to central axis 412. Thus, the sensing fields 432 and 434 cansense these spatial regions relative to the central axis 412 regardlessof the position of the central axis 412 of handle 402 in space, andregardless of the movement of handle 402 in its degrees of freedom. Forexample, if handle 402 can rotate about axis 435 at a rotary couplingwith another link (similar to axis 322 in FIG. 3 ), the sensing fields432 and 434 rotate with handle 402 about axis 435. Handle 402 may alsorotate about the central axis 412, and this rotation is relative to thecentral axis 412, sensing fields 432 and 434, and base member 408.

The size (e.g., volume and/or depth) of the sensing field of eachindividual sensor 440 and 442 is typically limited. The use of twosensing fields 432 and 434 allows the sensing field of the presencesensing system 430 to be extended to a larger total size (e.g., largevolume and/or larger range) than when using a sensing field of a singlesensor. The total sensed field in this implementation is extended tocover the regions on the sides of the central axis 412 of handle 402.

In some implementations, portions of the sensing fields 432 and/or 434can be blocked or adjusted in size or dimensions, e.g., by selectingparticular settings of the sensors emitting the sensing fields. In someimplementations, one or more of the sensors 440 and 442 may bephysically masked to block portions of the standard sensing field of thesensor from being sensed. For example, this can prevent the sensor fromdetecting objects such as grip members 406 or other components of handle402 which are to be ignored.

In some implementations using multiple sensing fields, as in the exampleof FIG. 4 using sensing fields 432 and 434, a hand (or other object) canbe detected in one of the sensing fields and may not be detected in theother sensing field(s). In some implementations, if a hand is detectedin one sensing field and not in the other sensing field, then adetection is made, e.g., detection of an object is considered to haveoccurred. In other implementations, a detection of a hand requires thatthe hand be detected in both (or all) sensing fields.

Detection of a hand by the presence sensing system can occur when thehand contacts the control input device, e.g., during operation of thecontrol input device by the hand, and/or can occur when the hand doesnot contact the control input device, e.g., on approach or departure ofthe hand from the control input device. In some implementations,multiple types of hand detection can be performed by the presencesensing system, e.g., a first type of detection of a hand approachingthe control input device, and a second type of detection of a hand in anoperating position where the control input device can be operated by thehand.

An advantage of the sensing fields 432 and 434 in the configuration ofFIGS. 4-6 is that these fields are oriented such that a user's handenters and/or is positioned within one or both of the sensing fields asthe hand approaches handle 402, e.g., to operate the handle. Thus,presence sensing system 430 can detect the presence of a hand before ithas contacted handle 402 (including contacting grips 406 or centralportion 407), and/or after it has released contact with handle 402. Insome implementations, the sensing system 430 can detect the hand before(or after) it contacts any part of the control input device portion 400.A system can use non-contact hand detection to, for example, enableparticular associated functions of the control input device (e.g.,powering up particular systems), track the hand to determine atrajectory of the hand and anticipate contact with the handle, etc.,some examples of which are described below.

The configuration of the sensing fields 432 and 434 as shown in FIGS.4-6 also or alternatively senses the presence of a hand operating thehandle 402. For example, while a hand is grasping the two grip members406, the hand extends into one or both sensing fields 432 and 434 on thesides and to the front of handle 402. This sensing field configurationcan be used with any control input device of the system without changesneeded for operation of handle 402 by a left hand or a right hand, sinceboth left and right sides of central axis 412 are sensed (e.g., handpresence may be more easily detected on a particular side of the centralaxis 412 depending on whether the left or right hand is operating handle402). The detection of the hand occurs at any rotational position ofhandle 402 about central axis 412, since a portion of the hand extendsinto one or both sides of handle 402 where sensing fields 432 and 434are present at any such rotational position. Furthermore, a portion ofthe sensing fields 432 and 434 extends below the distal end 405 ofhandle 402 as shown in FIG. 6 , allowing the sensors 440 and 442 tosense finger(s) (e.g., a thumb) of the user's hand which may extendbelow the central axis 412 in several positions of the handle 402 aboutcentral axis 412.

The sensing field(s) of described implementations are advantageouscompared to contact sensors or sensors detecting presence on or verynear a handle surface, since such sensors may not detect a hand whenfingers of the hand are lifted away from the grip members 406 (e.g.,within finger loops) during operation of handle 402, when fingers changeposition during operation (e.g., such that only finger tips or otherfinger portions contact the handle), and/or when fingers change positionand rotate handle 402 to a rotational position about axis 412 that isnot within the sensing range of the sensors. For example, in somecontrol system implementations, fingers of the hand may be adjusted bythe user to obtain a grasping position that causes the control inputdevice to match its position and orientation to a controlled manipulatordevice such as an instrument end effector. Contact sensors have to sensemany surfaces over a large surface area of handle 402 to sense suchdifferent hand positions, and such sensors may not be able to detectsome hand positions. Thus, a user could still be controlling themanipulator device and desire to stay in the controlling mode betweencontrol input device and manipulator device, but a system having suchcontact sensors may not sense the user's contact in cases as justdescribed and may deactivate the controlling mode.

In some implementations, detections from multiple sensors and/or sensingfields, such as sensing fields 432 and 434, can be used in combinationto detect a hand of a user. For example, measured distance and/orvelocity detected from multiple sensors can be used in variousimplementations for detection. In first example implementations,position values (e.g., distance values) describing a position of adetected object (such as a hand) in sensing fields can be used. Forexample, if both of two sensors 440 and 442 measure distance values of adetected object that meet a threshold (e.g., the object is detected at adistance from the reference location that is below a distancethreshold), then a hand is considered to be detected, else a hand is notdetected. In a different implementation, if one of the measured distancevalues meets the threshold, then a hand is considered to be detected,else a hand is not considered detected. In some implementations,different distance thresholds can be used. For example, a first distancethreshold that is closer to the handle can be used to detect that a handis contacting and/or operating the handle, and a second distancethreshold that is further away from the handle than the first distancethreshold can be used to detect nearby presence of the hand, e.g.,detecting a hand that is not contacting the handle, and/or detectingwhether the hand may be approaching or departing the handle, etc. Insome examples, detection of nearby presence can be used to alert thesystem that a hand may soon operate the handle, activate otherassociated functions of the system, etc., some examples of which aredescribed below.

In additional example implementations, the velocity of an object sensedby the sensors can be used. For example, if both of two sensors 440 and442 sense a hand and both sensors measure a velocity of the hand thatmeets (e.g., is below) a velocity threshold, then a hand is consideredto be detected, else a hand is not considered detected. In a differentimplementation, if both sensors sense a hand and one of the measuredvelocity values meets the threshold, then a hand is considered to bedetected, else a hand is not considered detected.

In additional example implementations, a combination of position (e.g.,distance) and velocity of an object sensed by the sensors can be used.For example, if both of two sensors 440 and 442 measure a distance thatmeets a distance threshold and a velocity that meets a velocitythreshold, then a hand is considered to be detected, else a hand is notdetected. In a different implementation, if both the sensors measure adistance that meets the distance threshold or a velocity that meets thevelocity threshold, then a hand is considered to be detected, else ahand is not detected. In a different implementation, if one of themeasured distance values meets the distance threshold and one of thevelocity values meets the velocity threshold, then a hand is consideredto be detected, else a hand is not detected. Other variations andpermutations can be used. In some implementations, more than two sensorsand/or sensing fields can be used, allowing additional combinations ofdistance and/or velocity detection (relative to thresholds) by all orsubsets of the sensors or sensing fields to be used to determine whethera hand is considered to be detected.

In some implementations using multiple (e.g., two) sensing fields, anobject (e.g., a hand) may be detected in one of the sensing fields andnot detected in another sensing field. In some implementations, adetection of a hand requires that the hand be detected in multiple (orall) sensing fields of the control input device. In otherimplementations, if a hand is detected in one sensing field and not inthe other sensing field, then a detection is considered to haveoccurred. For example, in some implementations, only one of the sensingfields may be able to detect a hand due to different hand posturesrelative to sensor field placement relative to the handle. In someexamples, on a right-hand control input device, a right sensor (e.g.,sensor 442 of FIG. 5 ) may detect the right hand and the left sensor(e.g., sensor 440 of FIG. 5 ) may not detect the right hand, andvice-versa for a left-hand control input device. If an object isdetected only by one particular sensor (e.g., the right sensor on theright-hand control input device, or the left sensor on the left-handcontrol input device), such a system can be configured to indicate thata hand detection has occurred.

In some implementations, additional sensors can be provided (e.g., twoor more right sensors 442 and two or more left sensors 440 on eachcontrol input device). In some examples, both right sensors must sensethe hand for it to be considered a hand detection. This allowsredundancy in the detection and sensor fault detection.

Sensors 440 and 442 are located on the parallel portion 422 of the basemember 408 and not on the distal end 405 of handle 402, allowing sensorelectronics to be more easily housed, powered, and/or communicated with(e.g., via physical connectors and/or wires) on the base member 408rather than on the rotating end of the smaller handle 402.

In various implementations, various types of sensors 440 and 442 can beused, e.g., non-contact sensors that sense an object in a sensing field.These sensors may provide more robust sensing than contact sensors insome implementations, e.g., they can sense a hand regardless of whetherthe hand is wearing a glove or is wet/dry, and they are more tolerant tonearby electric fields, magnetic fields, or energy output.

In various implementations, the sensors 440 and 442 can sense energyreflected by an object in the field (e.g., optical time of flight,reflected laser, or ultrasound sensors), sense energy radiated by anobject in a sensor field (e.g., heat energy in the infrared spectrum),or sense other physical quantities (e.g., physical pressure, electricalcapacitance change, etc.). The energy or other physical quantity can bedetected directly (e.g., an imaging camera) or indirectly by an effectit causes (e.g., a thermopile sensor).

For example, in some implementations, electromagnetic sensors (e.g.,optical sensors, infrared sensors, etc.) can be used, which are able todetect any of various ranges of wavelengths of electromagneticradiation, including visible light, infrared light, etc. In someexamples, an electromagnetic sensor includes an emitter that emits aelectromagnetic signal in the sensing field, and a detector that detectsthe electromagnetic signal (or a portion thereof) reflected from anobject in the sensing field. For example, sensors 440 and 442 can beoptical time-of-flight sensors that detect an object by measuring aposition of the object that is the distance between the sensor and theobject in the sensing field of the sensor, based on a measured timedifference between the emission of an electromagnetic signal and thereturn of the electromagnetic signal to the sensor after it has beenreflected by the object. Since the time-of-flight sensor can detect thedistance of a hand to the sensor, the system can determine the directionof movement of a hand by continually determining the distance of sensedobjects. In this way, the sensor can detect whether a hand isapproaching the handle 402 or is moving away from the handle 402. Insome implementations, this detection can be used to determine whetherthe user intends to operate the control input device, e.g., an approachdirection toward the handle indicates such an intent. In someimplementations, if it is determined that the user is not intending tooperate the control input device, then some system components are notprovided power (e.g., displays do not provide visual output, motors arenot powered, etc.) until such intent is detected via a detected handdirection toward the handle.

Furthermore, a velocity of the detected object can also be determined insome implementations. In some examples, velocity can be determined basedon a difference of detected positions (e.g., distances to referencelocation) of the object over time, indicating distance moved by theobject over time. For example, the velocity can be used to determineuser intent to operate the control input device. In some examples, thedirection of the detected object is indicated by the determinedvelocity, e.g., a positive velocity indicates that the object is movingaway from the control input device and a negative velocity indicatesmovement toward the control input device. If, for example, the detectedobject is moving away fast (e.g., a large positive determined velocitythat exceeds a velocity threshold), the system can determine to exitcontrolling mode based on the velocity, irrespective of position (e.g.,distance) of the object from the control input device or sensor. If thedetected object moves slowly away (e.g., a small positive determinedvelocity that is less than the velocity threshold), then a position(e.g., distance) of the object can also be used to determine whether toexit controlling mode (e.g., a position outside a distance threshold toa handle reference location can cause to exit controlling mode). In someexamples, a faster velocity can indicate stronger user intent, and/orcan be used to determine whether to continue to monitor hand positionsto determine such user intent. For example, if a slower velocity of theobject (e.g., hand) is detected and the hand is a threshold distanceaway from the reference location (e.g., a threshold distance from thesensor detecting the hand, from the handle, or from another referencelocation), the system can wait and sense more data related to objectposition or velocity before making a determination that the user intendsto operate the control input device.

In some implementations, one or more thermopile sensors can be used. Athermopile sensor includes a detector that detects infrared radiationemitted by objects located in the associated sensing field of thesensor. The sensor detects thermal changes, e.g., a differentialtemperature change, from the presence of objects of differenttemperatures in its sensing field. The infrared radiation emitted by thehand is typically much stronger (warmer) than other objects orcomponents that may be located in the sensing field, e.g., portions ofthe handle in some implementations. In some implementations, portions ofthe handle that may radiate greater amounts of heat, such as an amountof heat within a threshold range of the lowest estimate of heat from ahand, can be positioned external to the sensing field.

The sensing field of a thermopile sensor can be a cone or other shapesimilarly as described above. A thermopile sensor can be placed at anyof a variety of locations on the handle 402, base member 408, and/orother attached links. For example, one or more thermopile sensors can belocated at any of the locations of the presence sensors and provide thesensing fields shown in FIGS. 4-8 .

In some implementations, sensors 440 and/or 442 are thermal imagingcameras (e.g., thermographic cameras). For example, a thermal imagingcamera can sense infrared radiation from warmer temperatures located inthe sensing field of the camera, and provide data based on the sensedradiation that can be processed into two-dimensional (2D) images. Thus,the thermal imaging camera detects the presence of body portions such asa hand located in the sensing field of the thermal imaging camera. Insome implementations, the sensing field of the thermal imaging camera isdirected over a region that encompassed both sensing fields 432 and 434shown in FIGS. 4-6 . In some implementations, one or more components ofthe handle 402 can be located in the sensing field of the thermalimaging camera, such as the distal end 405 of the handle. For example,the heat from such components can be ignored by the system, and the heatemitted from an object such as a hand located in the sensing field isdistinguishable by typically being greater. In some implementations, adetected object that is not a component of the system and which isdetected to emit lower than a threshold amount of heat (as associatedwith a hand) is ignored as being an object that is not a hand. One ormore thermal imaging cameras can be located at any of the locations ofthe presence sensors and provide the sensing fields shown in FIGS. 4-8 .

A thermal imaging camera can sense and store successive frames orcaptured images of the sensing field, allowing the camera to sense ofthe direction of motion of sensed objects over time. By analyzing suchsuccessive frames, the system can determine whether a hand is movingtoward the handle 402 or away from the handle 402. Furthermore, avelocity of the detected object (which can include direction ofmovement) can also be determined in some implementations, similarly asdescribed above for the time of flight sensor.

In some implementations, an infrared sensor can emit an infrared beam inits sensing field at an object, and detect the beam reflecting from asurface of an object in the sensing field to detect the object. In someimplementations, an electromagnetic sensor can detect a magnitude of areflected beam of electromagnetic radiation to determine a distance tothe sensor of a surface which reflected the beam (e.g., the greater themagnitude, the smaller the distance to the object).

In some implementations, one or more ultrasonic sensors can be used inthe presence sensing system. An ultrasonic sensor emits an ultrasonicwave which to an object and is reflected from the object. The sensorreceives the reflected wave, and the distance from the sensor to theobject is determined based on the time of travel of the wave. In someimplementations, an ultrasonic sensor can detect magnitudes of reflectedsonic pulses to indicate distance of the sensor element from the sensoror sensor array (the lower the reflected magnitude, the longer thedistance). An ultrasonic sensor can be located at any of the positionson the control input device described herein in FIGS. 4-8 . In someimplementations, ultrasonic sensors may have larger, less well-definedsensing fields than optical time-of-flight sensors, such that a singlesensor can be used to sense a sensing field around the handle 402 todetect user presence, e.g., a sensing field that includes the sensingfields 432 and 434 shown in FIGS. 4-6 .

In some implementations, one or more contact sensors can be used, whichdetect the presence of the user's hand when the hand physically contactsthe sensor or a surface physically connected to the sensor. For example,capacitive or resistive sensors can be used, which measure the change incapacitance or resistance, respectively, on the control input devicewhen the hand contacts the sensor (or when the hand is very near to thesensor, e.g., for capacitive sensors). Some types of contact sensors candetect energy from the hand (e.g., infrared sensors that sense onlywithin contact range). In some examples, contact sensors can be locatedas close as possible to the distal end 405, and/or can be located onportions of grip members 406 of the controller portion 400. In someimplementations, a capacitive ring sensor can be provided around aportion of the controller portion 400. For example, the ring sensor canrun along the curved surface around the base portion 420 of the basemember 408, e.g., near sensor 460. Such a ring sensor senses a change incapacitance when the hand is very near the ring, e.g., when fingers ofthe hand contact grip members 406.

In some implementations, one or more sensors of the presence sensingsystem can perform processing on detection signals and provide processedsignals to a control circuit of the system (e.g., a processor). Forexample, a sensing system can detect the positions of a hand or otherobject over time and determine a direction of the object relative to areference location, and/or determine a velocity of the object, and sendparameters or values describing the direction and/or velocity to acontrol circuit.

One or more of the sensors described herein can be implemented using anintegrated circuit sensor that includes, for example, a sensing element,signal conditioner, analog to digital converter (ADC), math engine tocalculate sensed characteristics (e.g., sensed object temperature for athermopile sensor), etc.

FIG. 7 is a side elevational view of controller portion 400 of thecontrol input device including an example implementation of a presencesensing system in which the presence sensors are located at the distalend of the handle. In this example, presence sensor 450 as shown in FIG.4 is provided at the distal end 405 of handle 402.

Sensor 450 senses objects in a sensing field 702. In someimplementations, as shown in the example of FIG. 7 , sensing field 702is directed (e.g., away from handle 402) to a region of space that is atleast partially in front of the distal end 405 of handle 402 withrespect to viewpoint of a user operating handle 402. In someimplementations, sensor 450 detects a portion of a palm or othernon-finger portion of a hand within the sensing field 702. In someimplementations, sensor 450 detects fingers and/or other portions of ahand that are operating the handle, e.g., with fingers contacting gripmembers 406 (example shown in dashed lines in FIG. 7 ). In someimplementations, sensor 450 can detect the approach of the hand towardthe handle 412, e.g., sensing field 702 extends beyond the distal end405 to a region of space in front of the distal end 405 similarly asdescribed above for other implementations.

In various implementations, the sensing field 702 can be directed on oneside of a horizontal plane extending through the central axis, e.g.,above such a horizontal plane as shown in FIG. 7 , and central axis 412extends below the sensing field 702 without entering or intersecting thesensing field. In some implementations, the sensing field 702 can extendbelow such a horizontal plane, or the sensing field 702 can beintersected by such a horizontal plane.

In some implementations, the sensing field 702 can be centered along thecentral axis 412 with reference to the view shown in FIG. 5 . Forexample, the sensing field 702 can be bisected by a vertical planeextending through the central axis 412 orthogonally to the horizontalplane described above. In some implementations, the sensing field 702can be directed at least partially to one side (e.g., the left or right)of the vertical plane, with reference to the view shown in FIG. 5 . Forexample, the directed side is where a palm or other portion of the handis expected to consistently be when operating the control input device.In some implementations, a left or right direction can accommodate oneof a left-handed or right-handed use of the control input device, inwhich the operating hand is more reliably sensed on one side of thevertical plane.

The depth of the sensing field 702 (indicated by line 704) from thesensor 450 can be based on the type of sensor used. It extendssufficiently to detect a portion of a hand that is operating the controlinput device, e.g., with fingers engaged with grip members 406 in thisexample. In some implementations, portions of the sensing fields 432and/or 434 can be blocked or adjusted in size or dimensions, e.g., byselecting particular settings of the sensor 450.

The sensing field 702 has a position that is fixed with respect tohandle 402 and changes with respect to base member 408, due to thesensor 450 being positioned on the distal end 405 of handle 402 suchthat the sensor 450 rotates with the handle about axis 412.

In some implementations, multiple sensors can be provided at the distalend 405. For example, two sensors in the approximate location of sensor450 can provide left and right sensing fields on left and right sides ofaxis 412, similarly to the sensors 440 and 442 as shown in FIG. 5 .

Sensor 450 can be any of a variety of types of sensors, similarly asdescribed above with respect to FIG. 4 . For example, sensor 450 can bean electromagnetic time-of-flight sensor, thermopile sensor, thermalimaging camera, infrared sensor, ultrasonic sensor, etc.

In some implementations, sensing field 702 can be shaped as a cone. Forexample, the sensing field can have a particular width at the sensor 450and extend in width in a direction away from the sensor 450. In someimplementations, sensing field 702 can be shaped as a cylinder,rectangle, or other shape similarly as described above. In someimplementations, the cone shape can be made wider or narrower.

In some implementations, handle 402 is external to the sensing field702, e.g., handle 402 does not extend into the sensing field. In someimplementations, a portion of handle 402 can extend into the sensingfield 702, e.g., such that a portion of handle 402 intersects thesensing field. For some types of sensors, reflected signals caused bysuch components of handle 402 can be normalized such that such handlecomponents are ignored and new objects located within the sensing fieldare detected by the sensor 450.

The sensing field 702 in the configuration of FIG. 7 can be orientedsuch that a user's hand enters the sensing field as the hand approacheshandle 402, e.g., to operate the handle. Thus, presence sensing system430 can detect the presence of a hand before it has contacted handle 402(including contacting grips 406 or central portion 407). In someimplementations, the sensing system 430 can detect the hand before itcontacts any part of the controller portion 400.

The configuration of the sensing field 702 effectively senses thepresence of a hand operating handle 402. For example, while a hand isgrasping the two grip members 406, one or more portions of the hand,such as the palm, are present in sensing field 702. This sensing fieldconfiguration can be used with a control input device of the systemwithout changes needed for operation of handle 402 by a left hand or aright hand. The detection of the hand occurs at any rotational positionof the handle about axis 412, since a portion of the hand extends intosensing field 702 at any such rotational position.

The sensing field(s) of described implementations are advantageouscompared to contact sensors or sensors detecting presence on or verynear a handle surface, since such sensors may not detect a hand invarious operating conditions similarly as described.

FIG. 8 is a side elevational view of controller portion 400 of thecontrol input device including an example implementation of a presencesensing system in which one or more presence sensors are located at thebase member end of handle 402 of the controller portion.

In this example, sensor 460, also shown in FIG. 4 , is provided at thebase portion 420 of the base member 408 of handle 402. For example,sensor 460 can be located at or near a side of the base portion 420 thatis near to the proximal end 404 of handle 402, allowing the sensor 460to provide a sensing field that encompasses at least a portion of handle402. In this example, the sensor 460 is placed on top of base portion420 with reference to the view of FIG. 8 . In some implementations,sensor 460 can be placed on a different side of the base portion, e.g.,at a bottom of base portion 420, or a different side of base portion420.

Sensor 460 senses objects in a sensing field 802. In someimplementations, as shown in the example of FIG. 8 , sensing field 802is directed (e.g., toward the operating hand), to a region of space thatis located above a horizontal plane intersecting the central axis 412 ofhandle 402, with reference to the view of FIG. 8 . The central axis 412extends below the sensing field 802 without intersecting the sensingfield 802. In some implementations, sensor 460 detects fingers and/orother portions of a hand within the sensing field 802. In someimplementations, sensor 460 detects a portion of a hand operating thehandle, e.g., detecting fingers contacting one or more grip members 406(example shown in dashed lines in FIG. 8 ). In some implementations,sensor 460 can detect the approach of the hand toward handle 402, e.g.,sensing field 802 extends beyond the distal end 405 to a region of spacein front of the proximal end 405 and/or to the left and right of thecentral axis 412 similarly as described above for other implementations.

In some implementations, the sensing field 802 can extend below such ahorizontal plane, e.g., using sensor 460 placed on a top side of baseportion 420 to direct the field lower or downward (with reference to theview of FIG. 8 ), and/or by using a sensor placed on a bottom side, leftor right side, or edge of base portion 420. Such a lower sensing fieldcan detect fingers in a lower region, e.g., fingers (e.g., thumb)contacting or near the lower grip member 406 shown in FIG. 8 . In someimplementations, the sensing field 802 can be centered along the centralaxis 412 with reference to the view as shown in FIG. 5 . For example,the sensing field 802 can be bisected by a vertical plane extendingthrough the central axis 412 orthogonally to the horizontal planedescribed above. In some implementations, the sensing field 802 candirected at least partially to one side (e.g., the left or right) of thevertical plane, with reference to the view shown in FIG. 5 . Forexample, the directed side can be a region where a palm or other portionof a hand is expected to consistently be when operating the controlinput device. In some implementations, a left or right direction canaccommodate one of a left-handed or right-handed use of the controlinput device, in which the operating hand is more reliably sensed on oneside of the vertical plane.

The depth of the sensing field 802 (indicated by line 804) from thesensor 450 can be based on the type of sensor used. It extendssufficiently to detect a portion of a hand that is operating the controlinput device, e.g., with fingers engaged with grip members 406 in thisexample. In some implementations, portions of the sensing field 802 canbe blocked or adjusted in size or dimensions, e.g., by selectingparticular settings of the sensor 460.

The sensing field 802 has a spatial position that is fixed with respectthe central axis 412 of handle 402. Thus, the sensing field 802 cansense the same spatial region relative to the central axis 412regardless of the position of the central axis 412 of handle 402 inspace. The rotational orientation of handle 402 about central axis 412varies with respect to the sensor 460 and may cause different portionsof the handle 402 and the hand operating handle 402 to be present withinthe sensing field 802. The sensing field 802 can be made of sufficientlysize and width such that a portion of the hand is always present withinthe sensing field 802, regardless of rotational orientation of handle402 about central axis 412.

In some implementations, multiple sensors can be provided on the baseportion 420. For example, two sensors in the approximate location ofsensor 460 can provide left and right sensing fields on left and rightsides of axis 412, similarly to the sensors 440 and 442 as shown in FIG.5 . In some implementations, sensors can be provided at multiplelocations around the base portion 420 to cover different spatial regionsthat each include a different portion of handle 402.

Sensor 460 can be any of a variety of types of sensors, similarly asdescribed above with respect to FIG. 4 . For example, sensor 450 can bean electromagnetic time-of-flight sensor, thermopile sensor, thermalimaging camera, ultrasonic sensor, infrared sensor, etc.

In some implementations, sensing field 802 can be shaped as a cone. Forexample, the sensing field can have a particular width at the sensor 460and extend in width in a direction away from the sensor 460. In someimplementations, sensing field 802 can be shaped as a cylinder,rectangle, or other shape similarly as described above. In someimplementations, the cone shape can be made wider or narrower. In someimplementations, portions of the sensing field 802 (and/or sensing field702 described for FIG. 7 ) can be blocked or adjusted in size ordimensions, e.g., by selecting particular settings of the sensorsemitting the sensing fields, and/or physically masking portions of thesensor 460 to block portions of the standard sensing field of the sensorfrom being sensed. For example, this can prevent the sensor 460 fromdetecting objects such as grip members 406 or other components of handle402 which are to be ignored.

In some implementations, a portion of handle 402 can extend into thesensing field 802, e.g., such that a portion of handle 402 intersectsthe sensing field. For example, a portion of a grip member 406 canextend into the sensing field 802 as shown in FIG. 8 . For some types ofsensors, reflected signals caused by such components of handle 402 canbe sensed and normalized such that such handle components are ignoredand new objects located within the sensing field are detected by thesensor 460.

The sensing field 802 in the configuration of FIG. 8 can be orientedsuch that a user's hand enters the sensing field as the hand approacheshandle 402, e.g., to operate the handle. Thus, presence sensing system430 can detect the presence of a hand before it has contacted handle 402(including contacting grips 406 or central portion 407). In someimplementations, the sensing system 430 can detect the hand before itcontacts any part of the controller portion 400.

The configuration of the sensing field 802 effectively senses thepresence of a hand operating handle 402. For example, while a handgrasps the two grip members 406, one or more portions of the hand extendinto sensing field 802. This sensing field configuration can be usedwith a control input device of the system without changes needed foroperation of handle 402 by a left hand or a right hand.

Each control input device of a control system can include its ownpresence sensing system of any of the implementations described herein,allowing each control input device to activate and deactivate acontrolling mode independently of other controllers based on presencedetection.

The sensing field(s) of described implementations are advantageouscompared to contact sensors or sensors detecting presence on or verynear a handle surface, since such sensors may not detect a hand invarious operating conditions similarly as described.

FIG. 9 is a flow diagram illustrating an example method to detect thepresence of a user's hand to operate a control input device, in someimplementations. Method 900 can, for example, be performed by a controlsystem, e.g., an example teleoperated system in which the control inputdevice is included in a system that controls a manipulator device, e.g.,manipulator system 104 of FIG. 1 . In some implementations, the controlinput device is a component of a user control system, e.g., user controlsystem 102 of FIG. 1 . The control input device can be or include, forexample, a portion 300 or 400 of control input device 210 or 212, oranother control input device as described herein. In someimplementations, the method can be performed by a controller circuitcomponent coupled to the control input device. In some examples, thecontroller can include one or more processors, e.g., microprocessors orother control circuits, some examples of which are described below withreference to FIG. 11 .

A single control input device is referred to in method 900 forexplanatory purposes. Other implementations can use a control inputdevice having one or more features described herein with other types ofsystems, e.g., non-teleoperated systems, a virtual environment (e.g.,medical simulation) having no physical manipulator device and/or nophysical subject interacting with a physical manipulator device, etc.Multiple control input devices can be similarly processed as describedin method 900, e.g., both control input devices 210 and 212 of FIG. 2 .

In block 902, a non-controlling mode of the control system (e.g.,teleoperated system 100) is activated. The non-controlling mode can alsobe considered a “safe mode” in which the control input devices are notenabled to provide control signals to a controlled device such asmanipulator system 104 if the control input devices are manipulated bythe user. Thus, for example, the manipulator system is disconnected fromthe control input device for non-controlling mode, e.g., the manipulatorsystem is not being controlled by the control input device. For example,the control input devices 210 and 212 can be manipulated by a user innon-controlling mode which will not cause any controlled motion of theelements of the manipulator system 104.

In block 904, it is determined whether an object is detected in thesensing field(s) of the hand presence sensing system of a control inputdevice, e.g., presence sensing system 430 as described above. Forexample, the object may be an operator hand detected in the sensingfield(s) of the hand presence sensing system of the control inputdevice. Such detection can indicate that a user may be ready to startusing the control input device, e.g., to control a manipulator device.The object may also be detected by other sensing systems of the system.In some implementations, the hand presence sensing system may be able todetect whether the object is a hand or is a different object, e.g.,based on the magnitude of sensed temperature of the object being withina range of temperatures. In some of these cases, if the object is notdetected as a hand, it can be ignored. In some implementations, adetected object within a sensing field of a hand presence sensor isconsidered to be a hand.

In some implementations, an object is considered to be detected by thehand presence sensing system (e.g., user hand presence detected so thatblock 918 is performed, below) if the object is detected to be within aparticular sensing range, e.g., within a threshold distance to areference location associated with the control input device. Forexample, the reference location can be location on a handle (e.g.,handle 402) of the control input device, a location of one or moresensors of the hand presence sensing system, a surface of a finger grip,a defined point between two handle surfaces, etc. In someimplementations, the hand presence sensing system may detect an objectin its sensing field, but the object will be ignored for purposes ofmethod 900 (e.g., cannot qualify as a hand presence detection) unless itis within the particular distance to the reference location.

If an object has not been detected, then the method can return to block902 to continue the non-controlling mode of the control input device. Ifan object has been detected, then the method may continue to block 906,in which a direction of movement of the detected object can bedetermined (if appropriate). For example, if the presence sensing system430 includes a time-of-flight sensor, thermal imaging camera, or othertypes of sensors as described above, then motion of the detected objectcan be tracked over time and the direction of movement determined fromsensed data. In some implementations, the direction of the detectedobject can be indicated in a direction parameter used in the method,e.g., the direction parameter sent in signals to a control circuit. Insome implementations, a velocity of the detected object can bedetermined as described above (which can include direction of movementand/or magnitude/speed of movement). In some implementations, thevelocity can be used to determine whether to ignore the object forpurposes of method 900 unless it meets a particular velocity threshold(e.g., has a velocity above, or alternatively below, the threshold). Insome implementations, the velocity can be used to determine whether touse the position of the detected object (e.g., distance to control inputdevice) in determining whether user presence is detected, similarly asdescribed above. For example, fast velocity away from the control inputdevice can indicate that object position is not needed, while slowvelocity away from the control input device can indicate to examineobject position to help determine user intent.

In some implementations, a timer can be used to provide a time limit fora continuous time period in which the control system is receptive todetecting a user's hand operating a control input device handle andactivating controlling mode. For example, a timer can be started afteran object is detected in block 904. Some implementations can start atimer based on other or additional conditions, or at other stages of themethod 900.

In block 908, it is determined whether the detected movement of theobject is in one or more first designated directions relative to thehandle of the control input device. The first designated direction(s)are used to indicate potential user intent. For example, the designateddirection can be toward the handle (or a reference location on thehandle), such that the distance between object and handle decreases.Such a direction of movement can indicate that the user may be intendingto move his or her hand to grasp the control input device. For example,a vector of the movement of the object can be estimated based on theobtained sensor data describing the last few positions of the object,and if this vector is within a threshold range of directions, then it isconsidered to be moving toward the handle. Other first designateddirections can be used in some implementations, e.g., directions towardparticular input controls of the handle, etc.

If the object is not determined to be moving in a designated directionrelative to the handle (e.g., toward the handle), then the method cancontinue to block 909 in which it is determined whether to restart thedetection process. For example, if a timer was started upon detection ofan object after block 904 as described above, it is checked whether atimeout has occurred, e.g., a time period has expired, which indicatesto restart the process. In some examples, the timeout period can be in arange 3 to 10 seconds. In some implementations, if it is determined thatthe direction of the object is a second designated direction thatindicates user intent not to immediately use the handle, the detectionprocess can be restarted. For example, the second designated directioncan be away from the handle in a particular threshold range ofdirections (e.g., moving in a direction that is away from a referencelocation on the handle such that it increases the distancetherebetween). In some implementations, the direction of the object canbe determined based on detecting a distance from a sensor to the objectover time, e.g., with multiple measurements. In some implementations, adetermined velocity of the detected object can be used to assistdetermination of user intent to immediately use the handle, e.g., if ithas a velocity above a velocity threshold.

In some implementations, if the object leaves the sensing field(s) ofthe presence sensing system, the detection process can be restarted. Ifit is determined to restart the detection process, the method returns toblock 902. Otherwise, the method returns to block 906 to continue todetermine the direction of the object, e.g., to determine if itsmovement changes direction toward the handle.

If the object is determined to be moving toward the handle in block 908,then the method continues to block 910, in which one or more systemfunctions are activated. For example, one or more graphical userinterfaces (GUIs) of the control system may have been turned off and notdisplayed on display devices of the system (e.g., display screens,augmented reality displays, virtual reality displays, etc.) during thenon-controlling mode. Such displays can be activated in block 910 suchthat display objects or features of the GUIs are displayed. In someimplementations, features or objects of the GUIs may have been in adimmed state in the non-controlling mode, and are brightened in block910 from the prior dimmed state. In some implementations, activatedfunctions can include supplying power to one or more components of thesystem, such as motors of the control input device to provide forcefeedback and/or gravity compensation, motors of a manipulator devicethat are configured to move arms and/or instruments, cameras for viewinga operating site, lights for illuminating instruments and/or operatingsite, manipulator instrument functions (e.g., suction, irrigation,energy, etc.), etc. In some implementations, activated functions caninclude moving the handle to a particular starting position in theworkspace of the control input device, e.g., via control of motors onconnected linkages. In some example implementations, activated functionscan include moving all or part of other components of the control systemto starting positions via control of motors. Such components can includedisplay devices (e.g., screens, viewers, etc.), foot pedals, seats, etc.The method continues to block 912.

In block 912, it is determined whether the object (e.g., a hand of theuser) has been detected in an operating position of the control inputdevice (e.g., an “operating detection”). For example, the hand presencesensing system can detect whether the user's fingers and/or palm are ina position, or sufficiently close to a position, that allows the hand tooperate the control input device in the intended manner. In someimplementations, it is detected whether the hand is at a particularposition, such as within a threshold distance of a particular referencelocation (e.g., a location on a surface of the handle or adjacent to asurface of the handle, such as a surface of a grip member 406, or alocation of a sensor of the presence sensing system). As in the examplesof FIGS. 4-8 , the hand presence sensing system can detect whether thedetected portion of the hand is at or near a particular location withinthe sensing field(s) of the sensing system, e.g., palm portion of thehand in a position in front of the distal end 405, fingers positionedagainst the grip members 406, etc. In some implementations, the hand isconsidered to be in an operating position if a threshold amount orpercentage of the sensed field is occupied by a hand, or in someimplementations, if one or more particular locations within the sensedfield are occupied by the hand. In some implementations, the hand isconsidered to be in an operating in an operating position if it isdetected anywhere within the sensing field(s) of the hand presencesensing system, e.g., near the handle, contacting the handle, etc. Insome implementations, if a particular distance is checked in block 904as described above, the threshold distance of block 912 can be different(e.g., smaller, thus detecting the hand closer to the handle) than theparticular distance of block 904. In some implementations, multiplesensing fields of multiple individual sensors on the control inputdevice can be used to determine whether object detection has occurred,e.g., as described above with respect to FIG. 4 .

If the hand is not detected in an operating position, the methodcontinues to block 914 in which it is determined whether the detectionprocess should be restarted, similarly as described above for block 909.For example, it is determined whether a timeout has occurred, or if thedetected object is moving away from the handle. If the detection processis to be restarted, then the method returns to block 902 to continuenon-controlling mode. Otherwise, the method returns to block 910 tocontinue to activate system functions.

If the hand is detected in an operating position in block 912, then themethod may continue to block 916 in which it is determined whether therehas been additional detection by the system of the presence of the userin an operating position of the control input device and/or controlsystem. For example, the presence sensor 214 described above withreference to FIG. 2 can detect the presence of a user's head in aviewing recess 211 of user control system 102, indicating intended useof the control input devices 210 and 212 by the user. In someimplementations, the user is required to move the grip members 406 to aparticular position in the grip members' degree of freedom, e.g., tomatch a position or orientation of a manipulator instrument to becontrolled by the control input device (and provides other safetyfeatures). In some implementations, other or additional presence sensorscan be used to sense user presence. Such additional presence detectionacts as an additional check or safeguard in detecting a user beforeactivating controlling mode. In some implementations, if such additionalsensing is utilized, one or more thresholds used by the system to detecta user and activate controlling mode can be reduced to allow presencerequirements to be more easily satisfied. For example, thresholds ofobject detection used by the presence sensing system 430 and/or presencesensor 214 can be eased (e.g., magnitude of sensed signal, time fordetection to be verified, etc.), allowing quicker confirmation of userpresence. Some implementations can omit block 916.

If there is no additional detection of the user as determined in block916, the method continues to block 914 to check for a restart to thedetection process as described above. If additional detection of theuser is determined in block 916 (or additional detection is notimplemented), the method continues to block 918.

In block 918, a controlling mode of the system is activated. Controllingmode allows the manipulations of the control input device to controlfunctions of a controlled manipulator device. For example, in ateleoperated system, the manipulations of the control input device cancontrol corresponding motions, output functions (output of heat,electricity, etc.), and/or other functions of a manipulator device incontrolling mode, such as moving an end effector in space, opening jawsof the end effector, outputting heat or other energy from the endeffector, etc. In some implementations, controlling mode can beactivated for a corresponding component of the manipulator device thatis controlled by the control input device.

In some implementations, feedback output from one or more components ofthe system can indicate to the user that controlling mode is active andthat the control input device now controls a manipulator device, e.g.,the manipulator system 104. In some implementations, the output caninclude visual output from display devices, audio output from audiodevices, forces output on the control input device from motors, etc. Themethod continues to block 920.

In block 920, it is determined whether the user has stopped operatingthe control input device. This determination can be made in multipleways in various implementations. In some implementations, a change inthe presence of the hand is detected by the presence sensing system 430,and the system considers this change to indicate that the user hasstopped operating the control input device. In some examples, theindication that the user is ceasing operation of the control inputdevice can include a release of the user's touch or grip of the hand onthe handle of the control input device. The user's release of touch orgrip on the handle can be detected by the system based on the sensorsignals from the hand presence sensing system 430.

In some example implementations, the indication of ceasing operation canbe the hand (or a portion of the hand) moving out of the sensingfield(s) of the presence sensing system such that the presence sensingsystem 430 no longer detects the hand (or the portion of the hand). Inanother example, the indication can be detecting the hand (or portionthereof) in a location outside of a threshold distance or radius from areference location of the control input device (e.g., a referencelocation such as a location on the handle or a sensor of the handpresence sensing system). In another example, the indication can bedetecting movement of the hand of the user in a particular directionrelative to the handle or a reference location of the control inputdevice, e.g., in a direction away from the handle or away from areference location (e.g., within a threshold range of vector directionsaway from a reference location on the handle). A combination of theseindications and/or other indications can be used as a determination thatthe user has stopped operating the control input device.

Furthermore, other presence sensors (or other types of sensors) of thesystem can be used to detect a user stopping operation. For example, thepresence sensor 214 of FIG. 2 can sense whether the head of the user hasbeen removed from the viewing position. In some implementations, if anypresence sensor of the system no longer detects the presence of theuser, the user is determined to have stopped operating the control inputdevice. In some implementations, actions of a manipulator device (e.g.,removal of a surgical instrument from a defined operating area orworksite) may be used to indicate that the user has stopped operation.

If cessation of user operation of the control input device is notdetected, then the method continues to block 922 to continue providingthe controlling mode of the control system, and the method returns toblock 920 to continue checking for an indication that the user hasstopped operating the control input device. If, in block 920, it isdetected that the user has stopped operating the control input device,then the method returns to block 902 to activate the non-controllingmode of the system. For example, control of the manipulator device 104is disconnected from the control input device based on the detectionthat the user is no longer operating the control input device. At block902 and following blocks, the method can check for a user operating thecontrol input device as described above.

In some implementations, the system can enter additional or alternatestates upon detecting that the user has stopped operating the controlinput device (e.g., at block 920). For example, upon loss of detectionof the user's hand, a hold or pause of the controlling mode can be madeactive, such that, if the hand is again detected within a thresholdperiod of time, the controlling mode can be re-entered more easily(e.g., with lower thresholds as described above) than when restartingthe detection process from block 902. In some implementations, upon lossof detection of the user's hand, a power save mode of the system can beentered, and the power save mode can be exited when the hand is againdetected.

In various implementations, a time delay can be provided after aqualifying detection is made that causes the system to activatecontrolling mode and/or to activate non-controlling mode. For example,the time delay delays the activation of the controlling mode and/ornon-controlling mode. In some examples, upon determining in block 916that controlling mode should be activated (e.g., upon detection of handpresence and other user presence), the system waits for a delay of 1second (or 2 seconds, etc.) before controlling mode is made active. Asimilar delay can be provided after determining to activatenon-controlling mode, such that the activation of the non-controllingmode is delayed.

In some implementations, upon loss of detection of a user's hand atblock 920, the process may return to block 910 to activate (and/ordeactivate) one or more system functions, e.g., particular inputcontrols, settings, and/or inputs of the system can be made active.Examples of such system functions can include input devices on anarmrest of a user control system (e.g., an armrest touchscreen onarmrest 110) becoming active and receptive to user input, or a handle orswitch to control a position of a viewing device or display device tobecome active and receptive to user input. In some implementations,these functions and other functions that are not used duringmanipulation of the control input device can be disallowed ordeactivated (e.g., input to control these functions can be ignored)during detection of user hand presence at the control input device,e.g., at block 920. If the user's hand presence is no longer detected,then these functions can be made active. In some implementations, theparticular hand(s) having presence detected can be utilized to determinewhich controls are activated. For example, if only the user's left handis removed from a control input device so that the presence of the handis no longer detected, then particular input controls and/or functionsthat are accessible to the left hand are made active, but input controlsand/or functions accessible to only the right hand are not made active,since the presence of the right hand is still detected at a differentcontrol input device.

In some implementations, multiple control input devices may be used,e.g., each control device is simultaneously manipulated by a respectivehand of a user. In one example, multiple control input devices 210 and212 can be used as shown in FIG. 2 . In some of these implementations,each of the control input devices has a range of motion such that afirst control input device may be positioned within the sensing field(e.g., as described herein) of the presence sensing system of a secondcontrol input device. In some implementations, this positioning maycause a false positive detection of the first control input device bythe presence sensing system of the second control input device, e.g., amistaken detection of the presence of a user hand operating the secondcontrol input device. To reduce the occurrence of such false positivedetections, the position and/or orientations of each control inputdevice in its workspace can be tracked by a control block coupled to thecontrol input devices (e.g., control block 1110 of FIG. 11 ). Forexample, device sensors that detect the positions of the control inputdevices (e.g., sensors that detect relative positions or orientations oflinks of a kinematic chain) can be used to determine these positions andorientations of control input devices.

In some of these implementations, if a detection of an object isperformed in method 900 (e.g., in block 904 and/or block 912), then thesystem checks the current position(s) and/or orientation(s) of the othercontrol input device(s) to determine whether the detected object is acontrol input device that has been sensed by the presence sensingsystem. If it is determined that a control input device has been sensed,then the detection of the object is ignored, e.g., treated as if noobject has been detected. If it is determined that a control inputdevice has not been sensed, then the system examines the detected objectas described above for method 900. In some implementations, signals fromother sensors (such as head presence sensor 214) can be examined asanother factor to indicate whether to ignore the detected object. Forexample, if the user's head is detected by sensor 214, then thedetection of the object may be acknowledged and not ignored.

In some other implementations, if a control input device is determinedto be the detected object as described above, the detection of theobject is acknowledged and not ignored, but one or more other parametersor thresholds that are used to cause controlling mode to be activated inblock 918 can be made tighter or more strict, thus providing stricterconditions (and more certain detection) to detect user presence. Forexample, other presence sensors (e.g., head presence sensor 214) can beassigned a closer threshold distance or smaller sensing field todetermine detection of user presence. In another example, grip members406 can be required to be positioned in a smaller range of positions tomatch a position or orientation of a controlled manipulator instrument,in order to allow controlling mode to be active. Other examples ofsetting or adjusting detection parameters or other presence systemfeatures based on presence detection are described below with respect toFIG. 10 .

Various implementations can use different portions of the methodsdisclosed herein. For example, some implementations can perform blocks902, 904, 910, 918, 920, and 922 without other blocks; someimplementations can perform blocks 902, 904, 910, 912, 918, 920, and 922without other blocks; some implementations can perform one or moreblocks without block 904; etc.

The blocks described in the methods disclosed herein can be performed ina different order than shown and/or simultaneously (partially orcompletely) with other blocks, where appropriate. Some blocks can beperformed for one portion of data and later performed again, e.g., foranother portion of data. Not all of the described blocks need beperformed in various implementations. In some implementations, blockscan be performed multiple times, in a different order, and/or atdifferent times in the methods. For example, blocks 904, 908, 912,and/or 916 can be performed sequentially, at least partially at the sametime, or in a different order.

In another example implementation, a method includes activating acontrolling mode in response to detecting the hand presence of the user,and activating a non-controlling mode in response to detecting absenceof hand presence of the user. In some implementations, the controllingmode is activated only in response to both detecting the hand presenceof the user and detecting presence of the user by one or more otherpresence detection devices of the control system that includes thecontrol input device. For example, the other presence detection devicescan include a head presence sensor (e.g., as shown in FIG. 2 ), one ormore grips of the control input device that have been moved to aparticular position, etc.

FIG. 10 is a flow diagram illustrating an example method 1000 todetermine and/or adjust presence sensing features and/or other systemfeatures based on presence sensor data, according to someimplementations. In some implementations, method 1000 can be performedin a system that also performs method 900 of FIG. 9 , or alternatively,method 1000 (or various blocks or portions of method 1000) can beperformed with other methods or systems having suitable presencesensing.

Method 1000 can, for example, be performed for a control system, e.g.,an example teleoperated system or other control system in which thecontrol input device is included in a system that controls a manipulatordevice. In some implementations, the control input device is a componentof a user control system, e.g., user control system 102 of FIG. 1 . Thecontrol input device can be, for example, a portion 300 or 400 ofcontrol input device 210 or 212, or another control input device asdescribed herein. In some implementations, method 1000 can be performedby a circuit component coupled to the control input device. In someexamples, the controller can include one or more processors, e.g.,microprocessors or other control circuits, some examples of which aredescribed below with reference to FIG. 11 . A single control inputdevice is referred to in method 1000 for explanatory purposes. Otherimplementations can use a control input device having one or morefeatures described herein with other types of systems, e.g.,non-teleoperated systems, a virtual environment (e.g., medicalsimulation) having no physical manipulator device and/or no physicalsubject interacting with a physical manipulator device, etc. Multiplecontrol input devices can be similarly processed as described in method1000, e.g., both control input devices 210 and 212 of FIG. 2 .

In block 1002, a controlling mode of the control system (e.g.,teleoperated system 100) is active. The controlling mode can be similaras described with reference to FIGS. 9 and 11 , e.g., a mode in whichthe control input devices are enabled to provide control signals to acontrolled device such as manipulator system 104 as the control inputdevices are manipulated by the user. For example, the control inputdevices 210 and 212 can be manipulated by a user while controlling modeis active, which causes corresponding motion of elements of themanipulator system 104 and/or changes of other functions of themanipulator system 104.

In block 1004, a hand is detected by the hand presence sensing systemand a position of the hand is determined. For example, the position canbe a distance of the hand relative to a reference location of thecontrol input device as determined as described herein. In someexamples, the hand can be detected in one or more sensing fields asdescribed in various implementations herein, e.g., within a thresholddistance of a reference location of the control input device, and/orsatisfying other conditions to be considered a hand detection thatallows controlling mode to be activated as described herein. Thedistance of the hand can be determined as a distance between the hand(or a portion thereof) and the reference location, which can be an endof a handle, sensor, finger grip, or other location, etc. The distanceor other position can be determined as described herein for any ofvarious sensor types and implementations. In some implementations,movement and a direction of the movement of the hand can also bedetermined in block 1004 similarly as described above.

In block 1006, characteristics of forces output on the control inputdevice can be determined based on the hand position determined in block1004 and forces with such characteristics are output as appropriate. Theoutput forces include forces output in one or more degrees of freedom ofthe control input device by one or more actuators, e.g., motors. Forexample, the determined force characteristics can include a maximumforce (force limit) on the control input device. In someimplementations, if the hand is detected more than a threshold distanceaway from the reference location of the control input device, themaximum force can be set to a smaller magnitude than if the hand isdetected at a distance closer than the threshold distance. In someexamples, this threshold distance can be independent of (e.g., smallerthan) the presence sensing threshold distance described above. In someexamples, the maximum force can be gradually adjusted to any of multiplemagnitudes based on different distances of a detected hand, or can beset at one of two possible magnitudes based on the threshold distance,etc. In some implementations, if the hand is detected to be at longerdistances of the hand presence sensing range, it is more uncertainwhether the user is intending to continue operating the control inputdevice compared to hand locations at distances closer to the controlinput device. Therefore, the forces on the control input device arereduced for safety, e.g., so that the control input device is not movedas much by the forces away from the user if the user is detected to beusing the control input device by the hand presence sensing system butthe user is not grasping the control input device.

In additional examples, the determined force characteristics can includemagnitudes or gain of the output forces based on the hand positiondetermined in block 1004. For example, all of the forces output on thecontrol input device can be reduced in magnitude by a particularpercentage that is based on the sensed distance of the hand (or distanceof the hand that is greater than a particular distance threshold). Aforce ramping rate can be similarly based on the hand distancedetermined in block 1004, e.g., a rate at which forces are increased toa particular magnitude. Any one or more of the described forcecharacteristics can be determined and used in block 1006 in variousimplementations.

In block 1008, safety features of the control input device are adjustedbased on the hand position determined in block 1004. Some examples ofsafety features include techniques to detect particular patterns (e.g.,sequences) of motion, acceleration, changes in direction, etc. of thecontrol input device in one or more of its degrees of freedom. Thepatterns can indicate whether the user is actively controlling or notactively controlling the control input device (e.g., a control inputdevice moving or “floating” on its own would not match the pattern,would not move in particular degrees of freedom, etc.). The adjustmentof such safety features in block 1008 can include, for example, changingthe parameters (e.g., thresholds or limits) of the techniques based onthe detected position (e.g., distance) of the hand. For example, if thehand is sensed at longer distances (e.g., greater than a thresholddistance), parameters can be changed to require detection of shortermovements, higher accelerations, and/or more or greater changes indirection of the control input device in order to detect active usercontrol of the control input device that would continue activation ofthe controlling mode, as compared to shorter sensed distances of thehand (e.g., below the threshold distance). This reflects the uncertaintyof user intent at longer sensed distances of the hand, such thatincreased safety measures are provided.

Another example of a safety feature includes a limit for the velocity ofthe control input device in one or more degrees of freedom. For example,the velocity can be physically reduced by controlling one or more forceoutput devices (e.g., brakes, motors, etc.) that are coupled to thecontrol input device, to apply forces to slow the maximum allowedvelocity of the control input device in associated degrees of freedom.In some examples, a control system can monitor the velocity of thecontrol input device and can command output motor torque that isproportional to the velocity of the control input device to slow themotion of the control input device when the velocity is above athreshold. The controller and motor thus can provide damping to thecontrol input device to slow it down. For example, if the hand is sensedat longer distances (e.g., more than a threshold distance), the maximumvelocity can be lowered in comparison to shorter sensed distances of thehand (e.g., below the threshold distance). Some implementations can usea sensed direction and/or velocity of the hand in the determination ofsafety feature adjustment, e.g., based on whether the direction istoward or away from the control input device.

In block 1010, one or more detection parameters of other presencesensor(s) of the control system (e.g., a control console of ateleoperated system) are determined based on the hand positiondetermined in block 1004. The other presence sensors are independent andseparate from the hand presence sensing system that senses the presenceof a user's hand as described herein. Other presence sensors caninclude, for example, the head presence sensor 214 of FIG. 2 thatdetects a head of a user, and/or other presence sensors (e.g., a bodydetection sensor, foot detection sensor, etc.). Determining detectionparameters of such other presence sensors can include, for example,determining one or more thresholds of sensing, ranges of sensing, and/ordurations of sensing.

For example, if the hand presence sensing system detects a position of ahand, e.g., in a particular range (e.g., more than a threshold distancefrom the control input device), tighter or stricter range(s) and/orthreshold(s) of detection can be set for the head presence sensor 214 ascompared to when shorter distances are sensed for the hand (e.g.,outside the particular range or below the threshold distance). Thetighter ranges and/or thresholds cause the head presence sensor todetect user presence via the user's head under stricter conditions(e.g., with less tolerance), e.g., when the head is closer to or withina more precise location relative to the head presence sensor, and/or issensed for a longer duration (period of time), as compared to sensingwith looser ranges and/or thresholds. This can allow the other presencesensors to provide more certain user detection, e.g., to compensate whenthe hand presence sensing system may have detected user presence withless certainty, e.g., when the user's hand is further from the controlinput device and/or appears likely to disconnect from the control inputdevice. Multiple distance thresholds for the hand sensing can be used insome implementations, e.g., to set various associated values of thedetection parameters of the other presence sensor(s) based on the hand'sdetected position relative to the thresholds.

In block 1012, one or more detection parameters of the hand presencesensing system are determined based on detection by other presencesensor(s) of the control system (e.g., a user control system 102 of ateleoperated system 100). As in block 1010, the other presence sensor(s)can include, for example, the head presence sensor 214 of FIG. 2 and/orother presence sensors. For example, it can be determined whetheranother presence sensor has detected the user within one or morethresholds defining a particular presence range. In some examples, theparticular presence range can be close to bounds of the other presencesensor, e.g., detection within a particular distance range of a limit ofa sensed range, or within a particular distance range to a thresholdindicating detection. For example, the user's head may be detectedwithin a particular presence range that is at the limit of a range oflocations qualifying for presence detection. In some implementations,the particular presence range can be specified as a distance range orposition range defined by thresholds.

In response to user detection within the particular presence range ofthe other presence sensor(s), one or more detection parameters are setor adjusted for the hand presence sensing system. Such detectionparameters of the hand presence system can include, for example, thebounds of sensing, e.g., a threshold, range, direction, velocity, and/ora duration of sensing. In some examples, if a sensed position of theuser's head is in the particular presence range, e.g., more than athreshold distance from a reference location of the head presencesensor, one or more tighter/stricter ranges and/or thresholds of sensingcan be set for the hand presence sensing system compared to when sensedpositions of the head are outside the presence range, e.g., below thethreshold distance from the reference location of the head presencesensor.

The tighter ranges and/or thresholds of sensing cause the hand presencesensing system to detect user presence via the user's hand understricter conditions (e.g., with less tolerance). For example, thestricter conditions can include the hand being closer to the controlinput device, moving in a direction more directly toward the controlinput device, having a lower velocity (e.g., in a particular direction),and/or being sensed for a longer duration (period of time), as comparedto sensing with looser ranges and/or thresholds when the user's head (orother sensed user presence) is detected outside the particular presencerange. For example, this allows the hand presence sensing system toprovide more certain user detection when other presence detectors mayhave detected user presence with less certainty. Multiple thresholds forthe other presence sensors can be used in some implementations, e.g., toset various values of the detection parameters of the hand presencesensing system based on user presence detected relative to thethresholds.

In some implementations, any one or more of the above blocks 1006, 1008,1010, and/or 1012 can use alternative or additional characteristics of adetected object to determine the presence sensing features described inthese blocks. For example, a determined direction of movement of thedetected hand, and/or a determined velocity of the hand, can be used inthe determination of output forces in block 1006 (e.g., by reducingoutput force magnitudes if the direction is away from the control inputdevice), safety features of block 1008, and presence detectionparameters of block 1010 and/or 1012. In various implementations, asensed pose of a hand (e.g., sensed orientation and position of the handin space) can be used in the determination of the presence sensingfeatures in any of the blocks 1006, 1008, 1010, and/or 1012; forexample, particular poses can be associated with particular forces ordetection parameters determined in blocks 1006, 1008, 1010, and/or 1012.

Various implementations can use different portions of the methodsdisclosed herein. For example, various implementations can perform anyone or more blocks of the set of blocks 1006, 1008, 1010, or 1012without performing one or more other blocks of this set. In someexamples, one implementation can perform only blocks 1002, 1004, and1006; another implementation can perform only blocks 1002, 1004, and1008; another implementation can perform blocks only 1002, 1004, and1010; and another implementation can perform only blocks 1002, 1004, and1012.

The blocks described in the methods disclosed herein can be performed ina different order than shown and/or simultaneously (partially orcompletely) with other blocks, where appropriate. Some blocks can beperformed for one portion of data and later performed again, e.g., foranother portion of data. Not all of the described blocks need beperformed in various implementations. In some implementations, blockscan be performed multiple times, in a different order, and/or atdifferent times in the methods.

FIG. 11 is a block diagram of an example control system 1100 which canbe used with one or more features described herein. System 1100 includesa master device 1102 that a user may manipulate in order to control aslave device 1104 in communication with the master device 1102. In someimplementations, master device 1102 can be, or can be included in, usercontrol system 102 of FIG. 1 . In some implementations, slave device1104 can be, or can be included in, manipulator system 104 of FIG. 1 .More generally, master device 1102 can be any type of device including acontrol input device (e.g., including portion 300 or 400 of a controlinput device) that can be physically manipulated by a user. Masterdevice 1102 generates control signals C1 to Cx indicating positions,states, and/or changes of one or more control input devices in theirdegrees of freedom. The master device 1102 can also generate controlsignals (not shown) to control block 1110 indicating selection ofphysical buttons and other manipulations by the user. The master device1102 can also generate control signals to control block 1110 includingdetection data associated with detection of user presence by one or moreuser presence sensing systems, e.g., a head presence sensing systemand/or a hand presence sensing system of the master device 1102 asdescribed herein (e.g., indication of hand detection, detectionparameters including distance, direction, and/or velocity of detectedobjects, etc).

A control block 1110 can be included in the master device 1102, in theslave device 1104, or in a separate device, e.g., an intermediary devicebetween master device 1102 and slave device 1104. In someimplementations, the control block 1110 can be distributed amongmultiple of these devices. Control block 1110 receives control signalsC1 to Cx and generates actuation signals A1 to Ay, which are sent toslave device 1104. Control block 1110 can also receive sensor signals B1to By from the slave device 1104 that indicate positions, orientations,states, and/or changes of various slave components (e.g., manipulatorarm elements). Control block 1110 can include general components such asa processor 1112, memory 1114, and interface hardware 1116 and 1118 forcommunication with master device 1102 and slave device 1104,respectively. Processor 1112 can execute program code and control basicoperations of the system 1100, including functions related to sensingthe switch mechanisms described herein, and can include one or moreprocessors of various types, including microprocessors, applicationspecific integrated circuits (ASICs), and other electronic circuits.Memory 1114 can store instructions for execution by the processor andcan include any suitable processor-readable storage medium, e.g., randomaccess memory (RAM), read-only memory (ROM), Electrical ErasableRead-only Memory (EEPROM), Flash memory, etc. Various other input andoutput devices can also be coupled to the control block 1110, e.g.,display(s) 1120 such as the viewer 213 of the user control system 102and/or display 124 of FIG. 2 . One or more other presence sensors 1122can provide signals to control block 1110 indicating detection of userpresence and/or parameters related to such detection, e.g., headpresence sensor 214 of FIG. 2 and hand presence sensor systems describedherein.

In this example, control block 1110 includes a mode control module 1140,a controlling mode module 1150, and a non-controlling mode module 1160.Other implementations can use other modules, e.g., a force outputcontrol module, sensor input signal module, etc. In someimplementations, the modules 1140, 1150, and 1160 can be implementedusing the processor 1112 and memory 1114, e.g., program instructionsstored in memory 1114 and/or other memory or storage devices connectedto control block 1110.

Mode control module 1140 can detect when a user initiates a controllingmode and a non-controlling mode of the system, e.g., by user selectionof controls, sensing a presence of a user at a user control system orcontrol input device, sensing required manipulation of a control inputdevice, etc. The mode control module can set the controlling mode or anon-controlling mode of the control block 1110 based on one or morecontrol signals C1 to Cx.

In some implementations, controlling mode module 1150 may be used tocontrol a controlling mode of control block 1110. Controlling modemodule 1150 can receive control signals C1 to Cx and can generateactuation signals A1 to Ay that control actuators of the slave device1104 and cause it to follow the movement of master device 1102, e.g., sothat the movements of slave device 1104 correspond to a mapping of themovements of master device 1102. Controlling mode module 1150 can alsobe used to control forces on the control input device of the masterdevice 1102, e.g., forces output on one or more components of thecontrol input device, e.g., grip members, using one or more controlsignals D1 to Dx output to actuator(s) used to apply forces to thecomponents, e.g., to the grip members of the control input device, in arotary degree of freedom of the control input device, on arm linkscoupled to the control input device, etc. In some examples, controlsignals D1 to Dx can be used to provide force feedback, gravitycompensation, etc.

In some implementations, a non-controlling mode module 1160 may be usedto control a non-controlling mode of system 1100. In the non-controllingmode, movement in one or more degrees of freedom of master device 1102,or other manipulations of master device 1102, has no effect on themovement of one or more components of slave device 1104. In someimplementations, non-controlling mode can include one or more otheroperating modes of the control block 1110, e.g., a selection mode inwhich movement of the control input device in one or more of its degreesof freedom and/or selection of the control switches of the control inputdevice can control selection of displayed options, e.g., in a graphicaluser interface displayed by display 1120 and/or other display device. Aviewing mode can allow movement of the control input device to control adisplay provided from cameras, or movement of cameras, that may not beincluded in the slave device 1104. Control signals C1 to Cx can be usedby the non-controlling mode module 1160 to control such elements (e.g.,cursor, views, etc.) and control signals D1 to Dx can be determined bythe non-controlling mode module to cause output of forces on the controlinput device during such non-controlling modes, e.g., to indicate to theuser interactions or events occurring during such modes.

Some implementations described herein, e.g., methods 900 and/or 1000,can be implemented, at least in part, by computer program instructionsor code which can be executed on a computer. For example, the code canbe implemented by one or more digital processors (e.g., microprocessorsor other processing circuitry). Instructions can be stored on a computerprogram product including a non-transitory computer readable medium(e.g., storage medium), where the computer readable medium can include amagnetic, optical, electromagnetic, or semiconductor storage mediumincluding semiconductor or solid state memory, magnetic tape, aremovable computer diskette, a random access memory (RAM), a read-onlymemory (ROM), flash memory, a rigid magnetic disk, an optical disk, amemory card, a solid-state memory drive, etc. The media may be or beincluded in a server or other device connected to a network such as theInternet that provides for the downloading of data and executableinstructions. Alternatively, implementations can be in hardware (logicgates, etc.), or in a combination of hardware and software. Examplehardware can be programmable processors (e.g. Field-Programmable GateArray (FPGA), Complex Programmable Logic Device), general purposeprocessors, graphics processors, Application Specific IntegratedCircuits (ASICs), and the like.

The functional blocks, operations, features, methods, devices, andsystems described in the present disclosure may be integrated or dividedinto different combinations of systems, devices, and functional blocks.

Although the present implementations have been described in accordancewith the examples shown, there can be variations to the implementationsand those variations are within the spirit and scope of the presentdisclosure. Accordingly, many modifications may be made withoutdeparting from the spirit and scope of the appended claims.

1. A control input device comprising: a base member; a handle coupled tothe base member and configured to be manually contacted at a gripportion of the handle and moved by a hand of a user in one or moredegrees of freedom; one or more control input sensors configured todetect positions or orientations of the handle in the one or moredegrees of freedom; and a presence sensor coupled to the base member,the presence sensor having a sensing field, and at least a portion ofthe sensing field being located proximate to the handle, wherein asignal generated by the presence sensor comprises a parameter, theparameter comprising a value that corresponds to: a direction of motionof the hand in the sensing field relative to the presence sensor, or avelocity of the hand in the sending field.
 2. The control input deviceof claim 1, wherein the presence sensor is configured to detectelectromagnetic radiation or an ultrasonic wave that is directed throughspace to the presence sensor by a presence of the hand in the sensingfield of the presence sensor.
 3. (canceled)
 4. (canceled)
 5. (canceled)6. (canceled)
 7. The control input device of claim 1, wherein thesensing field is shaped as, or approximately as, a cone that increasesin width in a direction away from the presence sensor.
 8. (canceled) 9.The control input device of claim 1, wherein: the presence sensor is afirst presence sensor configured to detect first electromagneticradiation, and the sensing field is a first sensing field located at afirst side of the handle; the control input device further comprises asecond presence sensor coupled to the base member the second presencesensor is configured to detect second electromagnetic radiation that isdirected through space to the second presence sensor by a presence ofthe hand in a second sensing field of the second presence sensor; andthe second sensing field is proximate to the handle and is located at asecond side of the handle that is opposite the first side.
 10. Thecontrol input device of claim 1, wherein: the parameter also indicates avariable distance between an object in the sensing field and thepresence sensor.
 11. The control input device of claim 1, wherein: thepresence sensor includes an electromagnetic sensor; the presence sensorincludes and emitter and a detector; the emitter is configured to emit afirst electromagnetic signal in the sensing field; and the detector isconfigured to detect the first electromagnetic signal reflected from thehand in the sensing field.
 12. The control input device of claim 11,wherein the presence sensor further includes an optical time-of-flightsensor that generates a signal comprising a value that corresponds to avariable distance between the hand in the sensing field and the presencesensor.
 13. The control input device of claim 1, wherein: the presencesensor includes a thermopile sensor or a thermal imaging camera; and thethermopile sensor or the thermal imaging camera includes a detectorconfigured to detect infrared radiation emitted by the hand in thesensing field.
 14. The control input device of claim 1, wherein: aportion of the handle includes a handle distal end, a handle proximalend opposite the handle distal end, and a central axis defined betweenthe handle distal end and the handle proximal end; the handle distal endis closer than the handle proximal end to the hand; a base portion ofthe base member includes a base distal end and a base proximal endopposite the base distal end; the base portion extends parallel orapproximately parallel to the central axis of the portion of the handle;and the presence sensor is located on the base distal end that is closerthan the base proximal end to the handle distal end.
 15. The controlinput device of claim 1, wherein: the handle includes a central portionthat extends along a central axis of the handle between a distal end anda proximal end of the handle; the handle includes two grip membersextending from the central portion; the two grip members are eachconfigured to be gripped by a corresponding finger of the hand; thecentral portion is configured to be positioned between at least twofingers of the hand during grip of the handle by the hand; and thesensing field is configured to cover a region including one or morefingers of the hand touching either of the two grip members.
 16. Thecontrol input device of claim 15, wherein: the one or more degrees offreedom include a roll degree of freedom; the handle is rotatable aboutthe central axis of the handle with respect to the base member in theroll degree of freedom; and the sensing field is configured to includeat least a portion of the hand at all orientations of the handle in theroll degree of freedom while the hand grips the handle.
 17. (canceled)18. (canceled)
 19. A control input device comprising: a handleconfigured to be manually contacted at a grip portion of the handle andmoved by a hand of a user in one or more degrees of freedom, wherein thehandle includes a central portion that extends along a central axis ofthe handle, and wherein the central portion is configured to bepositioned between at least two fingers of the hand during a grip of thehandle by the hand; one or more control input sensors configured todetect positions or orientations of the handle in the one or moredegrees of freedom; and a presence sensor coupled to a distal end of thehandle that is proximate to the hand, wherein the presence sensor isconfigured to detect electromagnetic radiation or an ultrasonic wavethat is directed through space to the presence sensor by a presence ofthe hand in a sensing field of the presence sensor, and wherein thesensing field is located proximate to the handle, wherein a signalgenerated by the presence sensor comprises a parameter, the parametercomprising a value that corresponds to: a direction of motion of thehand in the sensing field relative to the presence sensor, or a velocityof the hand in the sending field.
 20. (canceled)
 21. (canceled) 22.(canceled)
 23. The control input device of claim 19, wherein: whereinthe parameter additional indicates a variable distance between the handin the sensing field and the presence sensor.
 24. (canceled) 25.(canceled)
 26. (canceled)
 27. The control input device of claim 19,wherein: the presence sensor includes at least one of an electromagneticsensor, a thermopile sensor, or a thermal imaging camera; theelectromagnetic sensor includes an emitter and a detector, the emitterbeing configured to emit a first electromagnetic signal in the sensingfield, and the detector being configured to detect the firstelectromagnetic signal reflected from the hand in the sensing field; thethermopile sensor includes a detector configured to detect infraredradiation emitted by the hand in the sensing field; and the thermalimaging camera includes a detector configured to detect the infraredradiation emitted by the hand in the sensing field.
 28. A methodcomprising: activating a non-controlling mode in which a handle of acontrol input device is manually moveable by a user in one or moredegrees of freedom without moveably controlling a manipulator device incommunication with the control input device; in the non-controllingmode, sensing a presence of a hand of a user relative to the handle in asensing field of a presence sensor, a portion of the sensing field beinglocated proximate to the handle, wherein sensing the presence of thehand of the user includes; determining a direction of motion of the handrelative to the handle, or determining a velocity of the hand relativeto the handle; and in response to sensing the presence of the hand,activating a controlling mode of the control input device in which thehandle is moveable by the user in the one or more degrees of freedom tomoveably control the manipulator device.
 29. (canceled)
 30. (canceled)31. The method of claim 28, wherein: sensing the presence of the handincludes determining the velocity of the hand relative to the handle;the method further comprises: determining that the velocity of the handmeets a threshold velocity; and the activation of the controlling modeis performed in response to the velocity of the hand meeting thethreshold velocity.
 32. The method of claim 28, wherein: the methodfurther comprises activating the non-controlling mode in response tosensing an indication that the hand is no longer operating the handle;and the indication includes at least one of sensing the hand outside athreshold distance from the handle or sensing the hand moving in aparticular direction relative to the handle.
 33. (canceled)
 34. Themethod of claim 28, wherein: the method further comprises while in thecontrolling mode: determining a position of the hand relative to areference location of the control input device while in the controllingmode, and determining, based on the position of the hand, one or morecharacteristics of force to be output on the control input device; andthe one or more characteristics of force include at least one of: amaximum force magnitude output on the control input device, a gain offorce magnitude output on the control input device, or a rate at whichthe force magnitude on the control input device is increased.
 35. Themethod of claim 28, wherein: the method further comprises while in thecontrolling mode: determining a position of the hand relative to areference location of the control input device, and adjusting a safetyfeature of the control input device based on the position; and adjustingthe safety feature includes at least one of: changing parameters used indetection of patterns of motion, acceleration, or direction of thecontrol input device to detect active use of the control input device bythe user, or physically limiting a velocity of the control input devicein one or more degrees of freedom by using one or more force outputdevices coupled to a mechanism of the control input device.
 36. Themethod of claim 28, wherein: the method further comprises: detectingpresence of the user by one or more other presence sensors of thecontrol input device; and determining one or more detection parametersof a hand presence sensing system based on the detected presence of theuser by the one or more other presence sensors; the other presencesensors are independent and separate from the hand presence sensingsystem that performs the sensing of the presence of the hand; and theone or more detection parameters of the hand presence sensing systeminclude one or more of: a threshold of sensing, a range of sensing, or aduration of sensing.
 37. (canceled)
 38. (canceled)
 39. (canceled)